justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 237459.0@dunegpschedd02.fnal.gov

Jobsub ID237459.0@dunegpschedd02.fnal.gov
Workflow ID9598
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-04 19:30:27
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc01
Last heartbeat2025-11-05 00:32:30
From worker nodeHostnamewn-lb-15.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-04 19:31:11
Input filesmonte-carlo-009598-000001
JobscriptExit code0
Real time4h (17985s)
CPU time4h (17852s = 99%)
Max RSS bytes4433281024 (4227 MiB)
Outputting started2025-11-05 00:30:56
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09598/1/001/reco_stage1.root
Finished2025-11-05 00:32:30
Saved logsjustin-logs:237459.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 133726 traces, 0 summary
Retagger: tagging trace set: gauss with 101266 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 101266 traces tagged "gauss"
FrameSaver: q=3.37968e+06 n=1776569 tag=gauss
wclsFrameSaver: saving 133726 traces tagged "wiener"
FrameSaver: q=3.60403e+06 n=1570172 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 31639 XUs and 35016 XVs -> 489 XUVs
C:0 T:1 69523 XUs and 81231 XVs -> 7717 XUVs
C:0 T:2 240460 XUs and 269863 XVs -> 53228 XUVs
C:0 T:3 66266 XUs and 51625 XVs -> 1092 XUVs
C:0 T:4 8671 XUs and 9192 XVs -> 34 XUVs
C:0 T:5 27835 XUs and 21989 XVs -> 2000 XUVs
C:0 T:6 184189 XUs and 242516 XVs -> 51903 XUVs
C:0 T:7 35605 XUs and 39351 XVs -> 1091 XUVs
C:0 T:8 7684 XUs and 6392 XVs -> 93 XUVs
C:0 T:9 54506 XUs and 56629 XVs -> 3695 XUVs
C:0 T:10 126267 XUs and 127360 XVs -> 35267 XUVs
C:0 T:11 187 XUs and 128 XVs -> 18 XUVs
156627 XUVs total
12805 collection wire objects
146415 potential space points
Neighbour search...
56810905 tests to find 21112000 neighbours
Iterating with no regularization...
Begin: 7.00984e+07
0 4.93532e+07
1 4.8284e+07
2 4.81211e+07
3 4.8059e+07
4 4.80245e+07
Now with regularization...
Begin: 3.92549e+07
0 3.90877e+07
1 3.89631e+07
2 3.88689e+07
3 3.87917e+07
4 3.87309e+07
5 3.86812e+07
6 3.86416e+07
7 3.86079e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410313825
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410313825
fLifetime: 30000 [us]
05-Nov-2025 01:30:54 CET  Closed output file "reco_stage1.root"
05-Nov-2025 01:30:54 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          474.198       583.335       818.723       572.773       104.628        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00130967    0.00725018     0.013299      0.0070161    0.00355417       10     
reco:ophit:OpHitFinder                             0.405585      0.436104      0.497526      0.430615      0.0282114       10     
reco:opflash:OpFlashFinder                         0.249154      0.306864      0.403533      0.293645      0.0434518       10     
reco:opslicer:OpSlicer                             0.727836      0.875908       1.10529      0.885282      0.112756        10     
reco:caldata:DataPrepModule                         6.34886       6.59015       7.26281       6.48509      0.274238        10     
reco:wclsdatasp:WireCellToolkit                     45.9351       58.9094       67.9682       59.5952       6.95288        10     
reco:gaushit:GausHitFinder                          1.33746       1.61941       2.22491       1.49305      0.266188        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000521559   0.000873923   0.00137186    0.000758438   0.000263927      10     
reco:reco3d:SpacePointSolver                        13.4015       31.5172       96.4945       25.8282       22.8703        10     
reco:hitpdune:DisambigFromSpacePoints               1.54559       2.18744       3.41651       1.94519      0.584238        10     
reco:pandora:StandardPandora                        37.002        64.622        126.451       54.3105       28.8897        10     
reco:pandoraWriter:StandardPandora                 0.504982      0.602579       0.7499       0.583911      0.0833716       10     
reco:pandoraTrack:LArPandoraTrackCreation           9.40474       11.7712       18.5357       10.6368       2.58524        10     
reco:pandoraShower:LArPandoraShowerCreation         7.77565       14.0445       26.6916       13.2606       5.38044        10     
reco:pandoracalo:Calorimetry                        4.8204        6.65431       11.4116       6.39954       1.82674        10     
reco:pandoracalonosce:Calorimetry                   4.61637       6.10512       10.8915       5.71002        1.706         10     
reco:pandorapid:Chi2ParticleID                    0.00653377    0.00747961    0.00882684    0.00742133    0.000678944      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0478916     0.0629403     0.0866481     0.0598341     0.010032        10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00592655    0.00709681    0.00860381    0.00692575    0.000787822      10     
reco:pandoraShowercalo:ShowerCalorimetry            7.71612       13.4895       25.6239       12.5571       4.99229        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       7.61111       13.1809       26.477        12.5579       5.34319        10     
reco:emtrkmichelid:EmTrackMichelId                  207.427       244.957       324.967       236.869       33.4345        10     
reco:crttag:SingleCRTMatchingProducer              0.0137837     0.026996      0.0472112     0.0243647     0.0106346       10     
reco:crtreco:TwoCRTMatchingProducer                0.0122708     0.0240848     0.0370161     0.022673     0.00920266       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.862699       1.09256       1.44501       1.04919      0.162742        10     
reco:pandora2Track:LArPandoraTrackCreation          24.0123       35.3416       63.3495       32.9312       10.7482        10     
reco:pandora2calo:Calorimetry                       11.134        16.1543       23.4822       15.3092       3.56444        10     
reco:pandora2calonosce:Calorimetry                  11.2307       14.995        22.3121       14.1587       3.21103        10     
reco:pandora2pid:Chi2ParticleID                    0.0140308     0.016778      0.0208606     0.017149     0.00211137       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0637426     0.083229      0.106825      0.0819121     0.0118683       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0138086     0.0169788     0.0205451     0.0170498    0.00221185       10     
reco:pandora2Shower:LArPandoraShowerCreation        7.58199       11.283        19.1505       9.85276       3.55458        10     
reco:pandora2Showercalo:ShowerCalorimetry           8.57509       12.4156       20.5996       11.2504       3.57418        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      6.97359       9.8107        15.6834       8.73011       3.15069        10     
[art]:TriggerResults:TriggerResultInserter        3.7687e-05    6.01971e-05   0.000138897   5.06055e-05   2.85871e-05      10     
end_path:out1:RootOutput                           6.91e-06     9.2113e-06    1.6834e-05     8.332e-06    2.62751e-06      10     
end_path:out1:RootOutput(write)                     3.12645       3.81052       5.02259       3.73889       0.54593        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5232.36 MB
  Peak resident set size usage (VmHWM): 4433.28 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.30 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.06 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10839528
-rw-r--r--. 1 dune009 dune 2534142092 Nov  4 23:51 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2405246796 Nov  4 23:48 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2254089416 Nov  5 01:30 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1945709767 Nov  4 20:54 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1876320706 Nov  4 20:39 g4_stage1.root
-rw-r--r--. 1 dune009 dune   83486830 Nov  5 01:25 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     243156 Nov  5 01:30 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Nov  5 01:30 mem.db
-rw-r--r--. 1 dune009 dune      62376 Nov  5 01:30 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Nov  5 01:30 time.db
-rw-r--r--. 1 dune009 dune        519 Nov  4 23:51 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov  4 20:54 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov  5 01:30 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Nov  4 20:31 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Nov  5 01:30 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Nov  4 23:59 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Nov  4 23:51 debugprod.log
justIN time: 2026-02-11 07:38:56 UTC       justIN version: 01.06.00