justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301991.1@dunegpschedd01.fnal.gov

Jobsub ID301991.1@dunegpschedd01.fnal.gov
Workflow ID12492
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-28 16:18:52
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-01-28 20:26:25
From worker nodeHostnamewn2208251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-28 16:19:36
Input filesmonte-carlo-012492-000001
JobscriptExit code0
Real time4h (14716s)
CPU time3h (14219s = 96%)
Max RSS bytes4107096064 (3916 MiB)
Outputting started2026-01-28 20:24:54
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12492/1/001/reco_stage1.root
Finished2026-01-28 20:26:25
Saved logsjustin-logs:301991.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

unning tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 107193 traces, 0 summary
Retagger: tagging trace set: gauss with 83061 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 83061 traces tagged "gauss"
FrameSaver: q=3.02564e+06 n=1503008 tag=gauss
wclsFrameSaver: saving 107193 traces tagged "wiener"
FrameSaver: q=3.17984e+06 n=1337236 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 37621 XUs and 30201 XVs -> 379 XUVs
C:0 T:1 58099 XUs and 49214 XVs -> 3887 XUVs
C:0 T:2 112225 XUs and 143464 XVs -> 4734 XUVs
C:0 T:3 54699 XUs and 54394 XVs -> 548 XUVs
C:0 T:4 7936 XUs and 6002 XVs -> 38 XUVs
C:0 T:5 23004 XUs and 13621 XVs -> 2010 XUVs
C:0 T:6 97986 XUs and 93399 XVs -> 8968 XUVs
C:0 T:7 41328 XUs and 34813 XVs -> 786 XUVs
C:0 T:8 3168 XUs and 2872 XVs -> 23 XUVs
C:0 T:9 12827 XUs and 15199 XVs -> 931 XUVs
C:0 T:10 34796 XUs and 33369 XVs -> 7116 XUVs
C:0 T:11 148 XUs and 85 XVs -> 6 XUVs
29426 XUVs total
8931 collection wire objects
26319 potential space points
Neighbour search...
2333911 tests to find 1235112 neighbours
Iterating with no regularization...
Begin: 6.97672e+07
0 6.15135e+07
1 6.08685e+07
2 6.07864e+07
3 6.07634e+07
Now with regularization...
Begin: 5.15399e+07
0 5.15068e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 412020985
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 412020985
fLifetime: 30000 [us]
28-Jan-2026 20:24:50 GMT  Closed output file "reco_stage1.root"
28-Jan-2026 20:24:50 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          325.195       426.33        577.812       393.569       79.6841        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00144967    0.00494618     0.0133885    0.00420581    0.00331822       10     
reco:ophit:OpHitFinder                             0.214471      0.335426      0.415872      0.353702      0.0589054       10     
reco:opflash:OpFlashFinder                          0.1357        0.17563      0.203973      0.176306      0.0198302       10     
reco:opslicer:OpSlicer                             0.399433      0.793464       1.06028      0.856233      0.197379        10     
reco:caldata:DataPrepModule                         4.51537       6.77516       7.99403       7.43135       1.27271        10     
reco:wclsdatasp:WireCellToolkit                     31.3128       40.0382       47.0098       41.4003       5.10008        10     
reco:gaushit:GausHitFinder                         0.809424       1.38041       2.10518       1.34097      0.416106        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000380325   0.000611061   0.000961146   0.000607267   0.000173764      10     
reco:reco3d:SpacePointSolver                        9.06468       18.2907       29.1704       17.4205       7.0094         10     
reco:hitpdune:DisambigFromSpacePoints               1.23082       1.96052       3.02453       1.79985      0.654842        10     
reco:pandora:StandardPandora                        20.2085       39.2485       70.0547       34.0442       16.4906        10     
reco:pandoraWriter:StandardPandora                 0.380464      0.572112      0.829593      0.565339      0.138587        10     
reco:pandoraTrack:LArPandoraTrackCreation           3.29958       5.14938       7.33052       4.84894       1.23941        10     
reco:pandoraShower:LArPandoraShowerCreation         2.85658       5.51716       10.1177       5.20445       1.98982        10     
reco:pandoracalo:Calorimetry                        1.70843       2.92989       4.10976       2.93234      0.708743        10     
reco:pandoracalonosce:Calorimetry                   1.67801       2.62144       4.10437       2.47674       0.72538        10     
reco:pandorapid:Chi2ParticleID                    0.00401883    0.00702077     0.0110297    0.00781398    0.00230121       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0379884     0.0645734     0.0932609     0.0677142     0.020263        10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00408573     0.006568     0.00857978    0.00738015    0.00176675       10     
reco:pandoraShowercalo:ShowerCalorimetry            2.92925       5.56203       10.581        4.82468       2.18835        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.95517       5.42371       9.69847       5.03887       1.91945        10     
reco:emtrkmichelid:EmTrackMichelId                  189.934       229.356       303.683       217.818       33.5241        10     
reco:crttag:SingleCRTMatchingProducer              0.0149058     0.0216098     0.0375895     0.0206692    0.00611875       10     
reco:crtreco:TwoCRTMatchingProducer               0.00863555     0.0121247     0.0180169     0.0114845    0.00288669       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.663653       0.98678       1.36554      0.933856       0.19287        10     
reco:pandora2Track:LArPandoraTrackCreation          9.3818        17.1565       26.0113       17.3433        4.684         10     
reco:pandora2calo:Calorimetry                       4.64003       9.36228       15.4832       8.82832       3.13838        10     
reco:pandora2calonosce:Calorimetry                  4.08107       8.48442       15.5756       7.74967       3.12484        10     
reco:pandora2pid:Chi2ParticleID                   0.00993229     0.0163169     0.0223691     0.0179218    0.00519456       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0561087     0.0957022     0.179664      0.0889272     0.0379222       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00973246     0.0162974     0.0225946     0.0173024    0.00447434       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.16973       5.99617       10.8417       5.50044       2.07267        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.57258       7.59368       15.1401       6.82979       3.02341        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.03471       6.70071       13.1349       6.11365       2.60796        10     
[art]:TriggerResults:TriggerResultInserter        6.3599e-05    0.000137911   0.000417084   0.000101075   9.86018e-05      10     
end_path:out1:RootOutput                           1.079e-05    1.54462e-05    2.104e-05    1.4833e-05    3.29632e-06      10     
end_path:out1:RootOutput(write)                     2.44117       3.51378       4.78415       3.6763       0.731932        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4946.87 MB
  Peak resident set size usage (VmHWM): 4107.1 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.10 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.31 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.12 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9743660
-rw-r--r-- 1 dune01 dune01 2302112173 Jan 28 19:11 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2189195951 Jan 28 19:08 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1979133363 Jan 28 20:24 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1751709010 Jan 28 16:38 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1681514363 Jan 28 16:27 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   73214810 Jan 28 20:20 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242582 Jan 28 20:24 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Jan 28 20:24 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Jan 28 20:24 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Jan 28 20:24 time.db
-rw-r--r-- 1 dune01 dune01        519 Jan 28 19:11 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 28 16:38 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 28 20:24 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Jan 28 16:19 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Jan 28 20:24 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Jan 28 19:20 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Jan 28 19:11 debugprod.log
justIN time: 2026-02-04 16:37:19 UTC       justIN version: 01.06.00