justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 242133.0@dunegpschedd02.fnal.gov

Jobsub ID242133.0@dunegpschedd02.fnal.gov
Workflow ID9992
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-12 12:22:26
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-11-12 16:26:49
From worker nodeHostnamewn2207291.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-12 12:23:32
Input filesmonte-carlo-009992-000001
JobscriptExit code0
Real time3h (13827s)
CPU time3h (13365s = 96%)
Max RSS bytes4071002112 (3882 MiB)
Outputting started2025-11-12 16:14:00
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09992/1/001/reco_stage1.root
Finished2025-11-12 16:26:49
Saved logsjustin-logs:242133.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 109101 traces, 0 summary
Retagger: tagging trace set: gauss with 86637 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 86637 traces tagged "gauss"
FrameSaver: q=3.01423e+06 n=1570464 tag=gauss
wclsFrameSaver: saving 109101 traces tagged "wiener"
FrameSaver: q=3.16703e+06 n=1403915 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 24469 XUs and 19842 XVs -> 225 XUVs
C:0 T:1 55717 XUs and 42876 XVs -> 4733 XUVs
C:0 T:2 158989 XUs and 98215 XVs -> 16211 XUVs
C:0 T:3 45010 XUs and 38906 XVs -> 560 XUVs
C:0 T:4 7569 XUs and 6956 XVs -> 41 XUVs
C:0 T:5 36744 XUs and 23949 XVs -> 4335 XUVs
C:0 T:6 78706 XUs and 92067 XVs -> 5118 XUVs
C:0 T:7 30249 XUs and 39226 XVs -> 237 XUVs
C:0 T:8 13139 XUs and 11732 XVs -> 165 XUVs
C:0 T:9 100266 XUs and 106520 XVs -> 9737 XUVs
C:0 T:10 14293 XUs and 11246 XVs -> 2186 XUVs
C:0 T:11 155 XUs and 145 XVs -> 40 XUVs
43588 XUVs total
9929 collection wire objects
39497 potential space points
Neighbour search...
7065859 tests to find 3201304 neighbours
Iterating with no regularization...
Begin: 5.16974e+07
0 4.55151e+07
1 4.51117e+07
2 4.50511e+07
3 4.50331e+07
Now with regularization...
Begin: 3.67736e+07
0 3.67405e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410468598
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410468598
fLifetime: 30000 [us]
12-Nov-2025 16:13:57 GMT  Closed output file "reco_stage1.root"
12-Nov-2025 16:13:57 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          305.094       407.966       627.778       390.092       94.1316        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00168597    0.00513542    0.00723356    0.00559747    0.00191342       10     
reco:ophit:OpHitFinder                             0.255295      0.336967       0.42538      0.344062      0.0592404       10     
reco:opflash:OpFlashFinder                         0.134778      0.189791       0.31636      0.178519      0.0472109       10     
reco:opslicer:OpSlicer                             0.547956      0.747435       1.0492       0.716921      0.148475        10     
reco:caldata:DataPrepModule                         4.82059       6.80416       8.82651       6.56609       1.11189        10     
reco:wclsdatasp:WireCellToolkit                     35.8555       44.6633       68.2485       39.5966       11.0565        10     
reco:gaushit:GausHitFinder                         0.833086       1.36381       2.18903       1.30851      0.436358        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00042122    0.000809757   0.00185767    0.000706184   0.000429596      10     
reco:reco3d:SpacePointSolver                        9.81129       19.5605       30.2193       18.1825       5.78319        10     
reco:hitpdune:DisambigFromSpacePoints               1.26681       2.00874       2.96263       1.86102      0.535424        10     
reco:pandora:StandardPandora                        21.5625       35.5682       65.3734       29.7655       13.978         10     
reco:pandoraWriter:StandardPandora                 0.371867      0.595053       1.0214       0.540709      0.181872        10     
reco:pandoraTrack:LArPandoraTrackCreation           3.0978        5.75902       14.8178       5.08859       3.25647        10     
reco:pandoraShower:LArPandoraShowerCreation         3.01951       6.17895       16.687        5.14782       3.74098        10     
reco:pandoracalo:Calorimetry                        1.53097       3.39375       8.51873       3.13703       1.81945        10     
reco:pandoracalonosce:Calorimetry                   1.52126       3.18665       7.84145       2.9332        1.6805         10     
reco:pandorapid:Chi2ParticleID                    0.00396407    0.00786901     0.0179797    0.00750942    0.00380535       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0380213     0.0744668     0.106406      0.0776764     0.0204885       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00401602    0.00720122     0.0129419    0.00728159    0.00237662       10     
reco:pandoraShowercalo:ShowerCalorimetry            3.10377       6.84518       18.2563       5.65526       4.28335        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.78493       6.7037        16.9579       5.2747        4.0202         10     
reco:emtrkmichelid:EmTrackMichelId                  168.139       209.259       267.563       199.232       31.8206        10     
reco:crttag:SingleCRTMatchingProducer              0.0130302      0.02438      0.0535109     0.0201942     0.0117002       10     
reco:crtreco:TwoCRTMatchingProducer               0.00740959     0.0148563     0.0246605     0.0153372    0.00471638       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.610831       1.02987       1.43847       1.04655      0.229983        10     
reco:pandora2Track:LArPandoraTrackCreation          9.41526       16.1623       33.3413       13.2298       6.88697        10     
reco:pandora2calo:Calorimetry                       4.38902       8.34511       18.7375       6.88039       4.26295        10     
reco:pandora2calonosce:Calorimetry                  3.89227       7.79172       18.0957       6.85578       4.24801        10     
reco:pandora2pid:Chi2ParticleID                   0.00857958     0.0172961     0.0250582     0.0174361     0.0050686       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0477983     0.0907724     0.121236      0.0930811     0.0259329       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00856303     0.0166997     0.0236864     0.0156475    0.00502797       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.25693       5.61887       12.1212       4.60139       2.65952        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.49412       6.43324       11.2144       5.82184       2.41973        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.93527       5.73484       10.8592       5.43254       2.3468         10     
[art]:TriggerResults:TriggerResultInserter        7.0242e-05    0.000104373   0.000222447   9.1356e-05    4.19398e-05      10     
end_path:out1:RootOutput                           9.879e-06    1.81131e-05   4.9112e-05    1.28995e-05   1.16516e-05      10     
end_path:out1:RootOutput(write)                     2.23347       3.22104       4.55396       3.11389      0.770524        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4928.14 MB
  Peak resident set size usage (VmHWM): 4071 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.14 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.34 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.96 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.16 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9740980
-rw-r--r-- 1 dune01 dune01 2303942179 Nov 12 15:03 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2188908920 Nov 12 15:00 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1974159641 Nov 12 16:13 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1749173777 Nov 12 12:46 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1687333080 Nov 12 12:32 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   70615400 Nov 12 16:09 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242771 Nov 12 16:13 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Nov 12 16:13 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Nov 12 16:13 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Nov 12 16:13 time.db
-rw-r--r-- 1 dune01 dune01        519 Nov 12 15:03 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 12 12:46 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 12 16:13 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Nov 12 12:23 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Nov 12 16:13 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Nov 12 15:13 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Nov 12 15:03 debugprod.log
justIN time: 2026-02-10 01:17:42 UTC       justIN version: 01.06.00