justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263808.0@dunegpschedd01.fnal.gov

Jobsub ID263808.0@dunegpschedd01.fnal.gov
Workflow ID11031
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-05 12:33:42
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-12-05 18:32:03
From worker nodeHostnamewn1207241.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-05 12:35:23
Input filesmonte-carlo-011031-000001
JobscriptExit code0
Real time5h (21291s)
CPU time5h (20472s = 96%)
Max RSS bytes4154060800 (3961 MiB)
Outputting started2025-12-05 18:30:16
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11031/1/001/reco_stage1.root
Finished2025-12-05 18:32:03
Saved logsjustin-logs:263808.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

l adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 124863 traces, 0 summary
Retagger: tagging trace set: gauss with 95996 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 95996 traces tagged "gauss"
FrameSaver: q=3.23579e+06 n=1709452 tag=gauss
wclsFrameSaver: saving 124863 traces tagged "wiener"
FrameSaver: q=3.41229e+06 n=1515805 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 23949 XUs and 31685 XVs -> 198 XUVs
C:0 T:1 46763 XUs and 51914 XVs -> 2921 XUVs
C:0 T:2 75775 XUs and 267750 XVs -> 9024 XUVs
C:0 T:3 35734 XUs and 44362 XVs -> 524 XUVs
C:0 T:4 7936 XUs and 13778 XVs -> 227 XUVs
C:0 T:5 38276 XUs and 43442 XVs -> 3728 XUVs
C:0 T:6 125256 XUs and 148623 XVs -> 5156 XUVs
C:0 T:7 52575 XUs and 53951 XVs -> 444 XUVs
C:0 T:8 8953 XUs and 7498 XVs -> 65 XUVs
C:0 T:9 49077 XUs and 62497 XVs -> 4311 XUVs
C:0 T:10 25506 XUs and 13856 XVs -> 3324 XUVs
C:0 T:11 90 XUs and 65 XVs -> 18 XUVs
29940 XUVs total
11413 collection wire objects
27519 potential space points
Neighbour search...
2246037 tests to find 1071596 neighbours
Iterating with no regularization...
Begin: 5.78928e+07
0 4.52592e+07
1 4.46097e+07
2 4.45234e+07
3 4.44975e+07
Now with regularization...
Begin: 3.55699e+07
0 3.55483e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410931607
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410931607
fLifetime: 30000 [us]
05-Dec-2025 18:30:06 GMT  Closed output file "reco_stage1.root"
05-Dec-2025 18:30:06 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          558.763       722.385       851.153       728.215       79.4509        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00203336     0.0159391     0.0826893    0.00628607     0.0242634       10     
reco:ophit:OpHitFinder                             0.354052      0.454685      0.671488      0.445947      0.0812616       10     
reco:opflash:OpFlashFinder                         0.209375       0.26866      0.436067      0.251692      0.059361        10     
reco:opslicer:OpSlicer                             0.906049       1.15666       1.6013        1.13229      0.223592        10     
reco:caldata:DataPrepModule                         7.5151        9.12024       10.0777       9.19987      0.743725        10     
reco:wclsdatasp:WireCellToolkit                     59.9076       69.9016       76.5832        69.84        4.46885        10     
reco:gaushit:GausHitFinder                          1.62902       2.19106       2.64415       2.24042      0.270033        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000900538   0.00372749     0.0268805    0.00116028    0.00771953       10     
reco:reco3d:SpacePointSolver                        20.1796       28.9134       49.2434       28.0607       7.99168        10     
reco:hitpdune:DisambigFromSpacePoints               1.76332       2.81069       4.02177       2.79314      0.641863        10     
reco:pandora:StandardPandora                        34.3952       63.0668       119.391       58.3962       21.1439        10     
reco:pandoraWriter:StandardPandora                 0.567874      0.806527       1.00796      0.834729      0.135396        10     
reco:pandoraTrack:LArPandoraTrackCreation           6.60476       10.7249       17.5304       10.2136       2.65608        10     
reco:pandoraShower:LArPandoraShowerCreation         6.85288       11.555        19.3403       11.1006       3.32844        10     
reco:pandoracalo:Calorimetry                        3.45774       6.53528       11.0721       6.25134       1.77036        10     
reco:pandoracalonosce:Calorimetry                   3.10887       5.87373       7.92177       5.77069       1.27173        10     
reco:pandorapid:Chi2ParticleID                    0.00881805     0.0117534     0.0148241     0.0115727    0.00202583       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0744794      0.10397      0.129309      0.0998044     0.0169633       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00700428     0.0112929     0.0155836     0.0112309    0.00248723       10     
reco:pandoraShowercalo:ShowerCalorimetry            6.41311       12.3037       16.7571       12.8428       2.64703        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       4.70791       11.6571       15.6008       12.5296       3.14272        10     
reco:emtrkmichelid:EmTrackMichelId                  323.34        382.731       426.117       387.988       31.2432        10     
reco:crttag:SingleCRTMatchingProducer              0.015142      0.0613133     0.362964      0.0278274     0.101077        10     
reco:crtreco:TwoCRTMatchingProducer                0.0136962     0.0220294     0.0408263     0.0190289    0.00872048       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.899918       1.39578       1.76033       1.45404      0.279178        10     
reco:pandora2Track:LArPandoraTrackCreation          17.7756       28.9477       35.1158       29.7658       5.21634        10     
reco:pandora2calo:Calorimetry                       8.28809       16.3399       21.5961       17.0686       3.62679        10     
reco:pandora2calonosce:Calorimetry                  7.98599       14.5515       17.6307       14.9964       2.53215        10     
reco:pandora2pid:Chi2ParticleID                    0.0228658     0.0406367     0.169797      0.026938      0.0430822       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.120609      0.141791      0.200551      0.134385      0.0216045       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0186032     0.0261686     0.0289531     0.0273541    0.00304541       10     
reco:pandora2Shower:LArPandoraShowerCreation         6.85         10.5731       12.7326       10.738        1.71425        10     
reco:pandora2Showercalo:ShowerCalorimetry           7.35556       13.1719       19.3076       12.506        3.6407         10     
reco:pandora2Showercalonosce:ShowerCalorimetry      6.37522       11.6455       18.4501       10.8721       3.68474        10     
[art]:TriggerResults:TriggerResultInserter        3.6248e-05    7.17323e-05   0.000176914   5.38565e-05   4.13693e-05      10     
end_path:out1:RootOutput                           6.533e-06    0.00150833     0.0150016    8.5815e-06    0.00449775       10     
end_path:out1:RootOutput(write)                     4.32247       4.9854        5.67554       5.01229       0.39839        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5002.68 MB
  Peak resident set size usage (VmHWM): 4154.06 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.38 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.82 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.23 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10104188
-rw-r--r-- 1 dune01 dune01 2376931427 Dec  5 16:26 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2258160682 Dec  5 16:21 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2057094886 Dec  5 18:30 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1830204108 Dec  5 13:07 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1750122582 Dec  5 12:45 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   73541950 Dec  5 18:20 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     243330 Dec  5 18:30 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Dec  5 18:30 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Dec  5 18:30 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Dec  5 18:30 time.db
-rw-r--r-- 1 dune01 dune01        519 Dec  5 16:26 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec  5 13:07 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec  5 18:30 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Dec  5 12:35 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Dec  5 18:30 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Dec  5 16:37 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Dec  5 16:26 debugprod.log
justIN time: 2026-02-07 19:11:00 UTC       justIN version: 01.06.00