justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 267896.0@dunegpschedd02.fnal.gov

Jobsub ID267896.0@dunegpschedd02.fnal.gov
Workflow ID11688
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-10 17:45:48
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2026-01-10 22:40:49
From worker nodeHostnamewn-da-09.gina.surf.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-10 17:46:42
Input filesmonte-carlo-011688-000001
JobscriptExit code0
Real time4h (17531s)
CPU time4h (17353s = 98%)
Max RSS bytes4788596736 (4566 MiB)
Outputting started2026-01-10 22:38:54
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11688/1/001/reco_stage1.root
Finished2026-01-10 22:40:49
Saved logsjustin-logs:267896.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

dEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 184457 traces, 0 summary
Retagger: tagging trace set: gauss with 139400 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 139400 traces tagged "gauss"
FrameSaver: q=5.08498e+06 n=2468909 tag=gauss
wclsFrameSaver: saving 184457 traces tagged "wiener"
FrameSaver: q=5.40242e+06 n=2190042 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 55645 XUs and 69187 XVs -> 934 XUVs
C:0 T:1 126863 XUs and 145204 XVs -> 10372 XUVs
C:0 T:2 97093 XUs and 66926 XVs -> 9999 XUVs
C:0 T:3 32145 XUs and 25058 XVs -> 110 XUVs
C:0 T:4 20553 XUs and 23873 XVs -> 256 XUVs
C:0 T:5 129197 XUs and 122023 XVs -> 13020 XUVs
C:0 T:6 136849 XUs and 188956 XVs -> 14846 XUVs
C:0 T:7 46666 XUs and 49700 XVs -> 814 XUVs
C:0 T:8 13815 XUs and 15275 XVs -> 222 XUVs
C:0 T:9 123753 XUs and 174979 XVs -> 15048 XUVs
C:0 T:10 108728 XUs and 62400 XVs -> 23292 XUVs
C:0 T:11 120 XUs and 113 XVs -> 17 XUVs
88930 XUVs total
17904 collection wire objects
82114 potential space points
Neighbour search...
18913616 tests to find 7850662 neighbours
Iterating with no regularization...
Begin: 1.04826e+08
0 8.68842e+07
1 8.54413e+07
2 8.51991e+07
3 8.51234e+07
Now with regularization...
Begin: 6.90851e+07
0 6.89592e+07
1 6.88761e+07
2 6.88156e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411660146
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411660146
fLifetime: 30000 [us]
10-Jan-2026 23:38:52 CET  Closed output file "reco_stage1.root"
10-Jan-2026 23:38:52 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          394.438       612.873       1001.47       536.106       181.587        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00118138    0.00620542     0.0111491    0.00609376    0.00326755       10     
reco:ophit:OpHitFinder                             0.300755      0.345947      0.380039      0.348997      0.0215904       10     
reco:opflash:OpFlashFinder                         0.180367      0.248177      0.309669      0.236171      0.0376277       10     
reco:opslicer:OpSlicer                             0.782594      0.854592      0.953428      0.863534      0.0466805       10     
reco:caldata:DataPrepModule                         5.36555       5.66984       6.0334        5.69716      0.169255        10     
reco:wclsdatasp:WireCellToolkit                     40.3894       69.4188       83.8514       79.2573       16.6101        10     
reco:gaushit:GausHitFinder                          1.19817       1.66929       2.08661       1.64271      0.268252        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000789173   0.00109898    0.00170025    0.00109653    0.000239349      10     
reco:reco3d:SpacePointSolver                        13.1331       28.5617       62.5693       25.299        13.5282        10     
reco:hitpdune:DisambigFromSpacePoints               1.23659       2.17061       3.22834       2.02135      0.659349        10     
reco:pandora:StandardPandora                        22.6794       93.1433       321.072       51.2596       86.9757        10     
reco:pandoraWriter:StandardPandora                 0.412845      0.619776       0.86493      0.593138      0.122416        10     
reco:pandoraTrack:LArPandoraTrackCreation           5.52485       8.93334       11.9191       9.08076       1.92356        10     
reco:pandoraShower:LArPandoraShowerCreation         5.20308       11.0109       17.5593       9.48964       4.04182        10     
reco:pandoracalo:Calorimetry                        3.67216       4.71153       6.1326        4.5785       0.835212        10     
reco:pandoracalonosce:Calorimetry                   2.50655       4.37387       6.6347         4.319        1.12427        10     
reco:pandorapid:Chi2ParticleID                    0.00695193    0.00791898    0.00850607    0.00798602    0.000468759      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0510176     0.0645622     0.0786334     0.061722      0.0092509       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00641486    0.00735968    0.00863524    0.00721592    0.000656827      10     
reco:pandoraShowercalo:ShowerCalorimetry            4.27059       10.6281       16.2666       10.6725       3.49778        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.74174       10.1259       15.7139       10.0879       3.48923        10     
reco:emtrkmichelid:EmTrackMichelId                  198.297       270.511       358.002       252.381       51.6594        10     
reco:crttag:SingleCRTMatchingProducer              0.0172146     0.0243851     0.0314572     0.0258196    0.00503563       10     
reco:crtreco:TwoCRTMatchingProducer                0.0112412     0.0170953     0.026828      0.015388     0.00453816       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.850753       1.08685       1.41424       1.03753      0.176093        10     
reco:pandora2Track:LArPandoraTrackCreation          14.3636       28.1913       41.507        25.2586       8.6721         10     
reco:pandora2calo:Calorimetry                       5.81096       13.4992       19.5391       12.4632       4.24473        10     
reco:pandora2calonosce:Calorimetry                  6.81359       13.2408       20.0902       12.0716       3.96003        10     
reco:pandora2pid:Chi2ParticleID                    0.0149467     0.0197094     0.0237119     0.0196639    0.00290578       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0673811     0.0871832     0.114477      0.0812523     0.0156858       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0156037     0.0198856     0.0241961     0.0198727    0.00295377       10     
reco:pandora2Shower:LArPandoraShowerCreation        6.21125       10.2211       16.1097       9.27689       3.01687        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.72741       10.5462       16.3042       9.44047       3.25447        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      4.20991       8.72496       13.7645       8.0792        2.82382        10     
[art]:TriggerResults:TriggerResultInserter        3.2292e-05    5.75283e-05   0.00016345    4.46095e-05   3.62765e-05      10     
end_path:out1:RootOutput                           7.093e-06    8.4392e-06    1.4517e-05     7.796e-06    2.06478e-06      10     
end_path:out1:RootOutput(write)                     2.79907       3.86188       5.09155       3.60299      0.738787        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5591.6 MB
  Peak resident set size usage (VmHWM): 4788.6 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.24 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.51 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.13 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11925932
-rw-r--r--. 1 dune009 dune 2731453812 Jan 10 21:54 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2597587283 Jan 10 21:51 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2467445567 Jan 10 23:38 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2206905241 Jan 10 19:11 g4_stage2.root
-rw-r--r--. 1 dune009 dune 2121043363 Jan 10 18:54 g4_stage1.root
-rw-r--r--. 1 dune009 dune   87105610 Jan 10 23:30 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     244077 Jan 10 23:38 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Jan 10 23:38 mem.db
-rw-r--r--. 1 dune009 dune      62376 Jan 10 23:38 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Jan 10 23:38 time.db
-rw-r--r--. 1 dune009 dune        519 Jan 10 21:54 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 10 19:11 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 10 23:38 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Jan 10 18:46 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Jan 10 23:38 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Jan 10 22:02 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Jan 10 21:54 debugprod.log
justIN time: 2026-02-06 11:42:45 UTC       justIN version: 01.06.00