justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 266397.0@dunegpschedd01.fnal.gov

Jobsub ID266397.0@dunegpschedd01.fnal.gov
Workflow ID11148
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-10 23:21:30
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-12-11 03:10:58
From worker nodeHostnamewn2208251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-10 23:23:01
Input filesmonte-carlo-011148-000001
JobscriptExit code0
Real time3h (13569s)
CPU time3h (13225s = 97%)
Max RSS bytes4430438400 (4225 MiB)
Outputting started2025-12-11 03:09:12
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11148/1/001/reco_stage1.root
Finished2025-12-11 03:10:58
Saved logsjustin-logs:266397.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

olBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 149386 traces, 0 summary
Retagger: tagging trace set: gauss with 116420 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 116420 traces tagged "gauss"
FrameSaver: q=4.11847e+06 n=2103270 tag=gauss
wclsFrameSaver: saving 149386 traces tagged "wiener"
FrameSaver: q=4.35198e+06 n=1872379 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 50559 XUs and 33651 XVs -> 695 XUVs
C:0 T:1 146278 XUs and 81127 XVs -> 10412 XUVs
C:0 T:2 103672 XUs and 215284 XVs -> 8415 XUVs
C:0 T:3 45207 XUs and 72214 XVs -> 1294 XUVs
C:0 T:4 10938 XUs and 6698 XVs -> 60 XUVs
C:0 T:5 28592 XUs and 14658 XVs -> 1570 XUVs
C:0 T:6 178078 XUs and 248342 XVs -> 12022 XUVs
C:0 T:7 58794 XUs and 68494 XVs -> 972 XUVs
C:0 T:8 7641 XUs and 7441 XVs -> 50 XUVs
C:0 T:9 38539 XUs and 44719 XVs -> 3300 XUVs
C:0 T:10 60297 XUs and 38779 XVs -> 7549 XUVs
C:0 T:11 161 XUs and 102 XVs -> 13 XUVs
46352 XUVs total
13385 collection wire objects
40326 potential space points
Neighbour search...
3994164 tests to find 1754106 neighbours
Iterating with no regularization...
Begin: 1.1787e+08
0 8.55696e+07
1 8.3534e+07
2 8.32682e+07
3 8.31911e+07
Now with regularization...
Begin: 7.13391e+07
0 7.12662e+07
1 7.12295e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411041220
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411041220
fLifetime: 30000 [us]
11-Dec-2025 03:09:10 GMT  Closed output file "reco_stage1.root"
11-Dec-2025 03:09:10 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          321.889       530.319       636.82        545.365       85.8567        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00111038     0.0054181     0.0130405    0.00537615    0.00301281       10     
reco:ophit:OpHitFinder                             0.288977      0.371053      0.459622      0.360888      0.0525972       10     
reco:opflash:OpFlashFinder                          0.17447      0.223943      0.268241      0.216998      0.0311342       10     
reco:opslicer:OpSlicer                             0.680514      0.957365       1.43427      0.951373      0.215914        10     
reco:caldata:DataPrepModule                         4.56992       6.0264        7.57993       6.04916       1.04715        10     
reco:wclsdatasp:WireCellToolkit                     32.174        41.2552       45.9448       42.0524       3.66137        10     
reco:gaushit:GausHitFinder                         0.894839       1.67844       2.04344       1.81213      0.350702        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00045775    0.000815479   0.00130931    0.000767282   0.000203025      10     
reco:reco3d:SpacePointSolver                        11.2769       30.0192       57.6172       28.1813       12.0635        10     
reco:hitpdune:DisambigFromSpacePoints               1.05857       2.63109       3.9113        2.49653      0.801789        10     
reco:pandora:StandardPandora                        21.6874       66.163        119.65        71.0008       25.0978        10     
reco:pandoraWriter:StandardPandora                 0.360906      0.589414      0.825458      0.536838      0.147752        10     
reco:pandoraTrack:LArPandoraTrackCreation           3.22253       8.07747       12.0593       8.00491       2.19003        10     
reco:pandoraShower:LArPandoraShowerCreation         3.00541       10.2338       14.7036       10.5135       3.20388        10     
reco:pandoracalo:Calorimetry                        1.74186       4.42045       6.54101       4.37853       1.30643        10     
reco:pandoracalonosce:Calorimetry                   1.93463       4.01192       6.08183       4.14772       1.11408        10     
reco:pandorapid:Chi2ParticleID                    0.00536522     0.0074468     0.0102048    0.00750949     0.0014941       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0453146     0.0703486     0.0896694     0.0743397     0.0174228       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00468927    0.00740809     0.010907     0.00719501     0.0020593       10     
reco:pandoraShowercalo:ShowerCalorimetry            3.06998       10.069        15.215        10.6256       3.24212        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.71595       9.49335       14.3501       9.77942       2.90572        10     
reco:emtrkmichelid:EmTrackMichelId                  184.33        247.353       299.167       252.906       29.7297        10     
reco:crttag:SingleCRTMatchingProducer              0.0152939     0.0227558     0.0386994     0.0205936    0.00708773       10     
reco:crtreco:TwoCRTMatchingProducer               0.00737829     0.0157653     0.0250847     0.014826     0.00585705       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.686267       1.09361       1.41274       1.09072      0.199982        10     
reco:pandora2Track:LArPandoraTrackCreation          9.63119       26.6136       35.5053       27.4508       7.67291        10     
reco:pandora2calo:Calorimetry                       6.0267        14.0954       20.0332       14.7688       3.83719        10     
reco:pandora2calonosce:Calorimetry                  5.58067       13.3588       19.8779       13.076        4.05154        10     
reco:pandora2pid:Chi2ParticleID                    0.0131075     0.0204825     0.0269149     0.0218884     0.0057589       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0582466     0.0982487     0.129323      0.109243      0.0246024       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0129881     0.0198769     0.0277366     0.0215013    0.00542956       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.94152       8.65221       12.5684       8.42889       2.4315         10     
reco:pandora2Showercalo:ShowerCalorimetry           3.73898       10.0483       14.3042       9.80409       2.78421        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.85222       8.69746       13.751        7.86992       2.88495        10     
[art]:TriggerResults:TriggerResultInserter        5.9442e-05    8.3408e-05    0.000165071   7.44655e-05   3.06293e-05      10     
end_path:out1:RootOutput                           9.218e-06    1.21559e-05   1.7914e-05    1.21075e-05   2.59376e-06      10     
end_path:out1:RootOutput(write)                     2.57193       3.72114       4.6026        3.94733      0.728668        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5270.59 MB
  Peak resident set size usage (VmHWM): 4430.44 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.27 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.67 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.15 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11223796
-rw-r--r-- 1 dune01 dune01 2615779152 Dec 11 01:38 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2480887180 Dec 11 01:36 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2359455923 Dec 11 03:09 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 2031178501 Dec 10 23:42 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1915747671 Dec 10 23:30 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   89487110 Dec 11 03:03 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     243313 Dec 11 03:09 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Dec 11 03:09 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Dec 11 03:09 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Dec 11 03:09 time.db
-rw-r--r-- 1 dune01 dune01        519 Dec 11 01:38 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec 10 23:42 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec 11 03:09 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Dec 10 23:23 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Dec 11 03:09 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Dec 11 01:46 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Dec 11 01:38 debugprod.log
justIN time: 2026-02-07 13:30:01 UTC       justIN version: 01.06.00