justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 295604.1@dunegpschedd02.fnal.gov

Jobsub ID295604.1@dunegpschedd02.fnal.gov
Workflow ID12409
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-26 18:54:27
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-01-27 00:16:54
From worker nodeHostnamewn2203301.tier2.hep.manchester.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-26 18:56:16
Input filesmonte-carlo-012409-000001
JobscriptExit code0
Real time5h (19137s)
CPU time5h (18785s = 98%)
Max RSS bytes4340334592 (4139 MiB)
Outputting started2026-01-27 00:15:14
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12409/1/001/reco_stage1.root
Finished2026-01-27 00:16:54
Saved logsjustin-logs:295604.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Running tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 127642 traces, 0 summary
Retagger: tagging trace set: gauss with 96723 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 96723 traces tagged "gauss"
FrameSaver: q=3.38157e+06 n=1692505 tag=gauss
wclsFrameSaver: saving 127642 traces tagged "wiener"
FrameSaver: q=3.58904e+06 n=1492148 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 25852 XUs and 30061 XVs -> 240 XUVs
C:0 T:1 52571 XUs and 50019 XVs -> 4768 XUVs
C:0 T:2 92921 XUs and 98914 XVs -> 3521 XUVs
C:0 T:3 47735 XUs and 39462 XVs -> 337 XUVs
C:0 T:4 14020 XUs and 14198 XVs -> 212 XUVs
C:0 T:5 62543 XUs and 43419 XVs -> 5783 XUVs
C:0 T:6 99061 XUs and 86471 XVs -> 3368 XUVs
C:0 T:7 46186 XUs and 36256 XVs -> 414 XUVs
C:0 T:8 8938 XUs and 7949 XVs -> 112 XUVs
C:0 T:9 75313 XUs and 75159 XVs -> 8316 XUVs
C:0 T:10 16814 XUs and 9351 XVs -> 1608 XUVs
C:0 T:11 6 XUs and 6 XVs -> 1 XUVs
28680 XUVs total
11066 collection wire objects
26353 potential space points
Neighbour search...
1239433 tests to find 678588 neighbours
Iterating with no regularization...
Begin: 6.9298e+07
0 5.76837e+07
1 5.67853e+07
2 5.66629e+07
3 5.66265e+07
Now with regularization...
Begin: 4.6319e+07
0 4.62823e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411982887
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411982887
fLifetime: 30000 [us]
27-Jan-2026 00:15:10 GMT  Closed output file "reco_stage1.root"
27-Jan-2026 00:15:10 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          499.503       638.876       815.358       636.476       101.401        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00185729    0.00452789     0.0064423     0.0047745    0.00133292       10     
reco:ophit:OpHitFinder                              0.27552      0.430315      0.536735      0.457942      0.0883329       10     
reco:opflash:OpFlashFinder                         0.181021      0.261165      0.318616      0.266234      0.0445591       10     
reco:opslicer:OpSlicer                             0.702551       1.04168       1.3781        1.03816      0.213858        10     
reco:caldata:DataPrepModule                         6.56437       8.67917       10.4474       8.89143       1.34341        10     
reco:wclsdatasp:WireCellToolkit                     58.3404       62.1217       67.2799       61.4797       2.62242        10     
reco:gaushit:GausHitFinder                          1.50837       1.8545        2.71722       1.64859       0.40164        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000587792   0.000825298   0.00114977    0.000807165   0.000158535      10     
reco:reco3d:SpacePointSolver                        12.5466       22.4168       36.0487       21.8409       6.4324         10     
reco:hitpdune:DisambigFromSpacePoints               2.29452       2.94605       3.93369       2.80878      0.523136        10     
reco:pandora:StandardPandora                        32.1073       49.5844       74.0474       47.5721       15.6601        10     
reco:pandoraWriter:StandardPandora                 0.499141      0.720625       1.03232      0.687797      0.151992        10     
reco:pandoraTrack:LArPandoraTrackCreation           6.83515       11.3372       16.3151       11.0939       2.99314        10     
reco:pandoraShower:LArPandoraShowerCreation         6.91927       12.5159       16.6579       12.6302       2.99475        10     
reco:pandoracalo:Calorimetry                        3.18287       5.08169       7.28355       5.03726       1.34931        10     
reco:pandoracalonosce:Calorimetry                   2.21526        4.619        7.2988        4.39043       1.46681        10     
reco:pandorapid:Chi2ParticleID                    0.00471007    0.00945313     0.0181074    0.00898415    0.00376421       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0590615     0.0964749     0.137196      0.0977289     0.0230933       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00461032    0.00790651     0.0115746    0.00721055    0.00241019       10     
reco:pandoraShowercalo:ShowerCalorimetry            4.94459       11.5435       16.566        11.0689       3.87824        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       5.79123       10.8577       15.5243       11.0395       3.64537        10     
reco:emtrkmichelid:EmTrackMichelId                  272.378       316.596       414.646       315.185       41.4813        10     
reco:crttag:SingleCRTMatchingProducer              0.0254488     0.0381267     0.0555872     0.0363798    0.00978161       10     
reco:crtreco:TwoCRTMatchingProducer                0.0202893     0.0305991     0.0473623     0.029787     0.00754499       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.23251       1.61221       1.93761       1.58549      0.232174        10     
reco:pandora2Track:LArPandoraTrackCreation          25.4533       37.3061       52.7327       36.4162       8.68007        10     
reco:pandora2calo:Calorimetry                       10.3953       16.2028       23.5416       16.4603       4.29809        10     
reco:pandora2calonosce:Calorimetry                  8.94155       14.9713       20.0246       15.9723       3.94733        10     
reco:pandora2pid:Chi2ParticleID                    0.0135315     0.0196704     0.0293267     0.018551     0.00439351       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0830082     0.115139      0.160165      0.108631      0.0230023       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0133277     0.018722      0.027018      0.0181201    0.00395016       10     
reco:pandora2Shower:LArPandoraShowerCreation        7.21613       12.2995       18.0214       11.8448       2.92966        10     
reco:pandora2Showercalo:ShowerCalorimetry           6.81156       15.3664       22.592        14.7888       4.77417        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      6.90517       13.6239       23.0007       11.2706       4.66075        10     
[art]:TriggerResults:TriggerResultInserter        7.7233e-05    0.000110824   0.000215602   0.000104545   3.79281e-05      10     
end_path:out1:RootOutput                           7.555e-06    1.03063e-05   1.6778e-05    9.8475e-06    2.49221e-06      10     
end_path:out1:RootOutput(write)                     2.79893       4.35891       5.98195       4.37054      0.973309        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5155.39 MB
  Peak resident set size usage (VmHWM): 4340.33 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.13 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.34 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.34 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.30 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10309184
-rw-r--r-- 1 dune01 dune01 2407751696 Jan 26 22:25 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2289781376 Jan 26 22:22 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2088605259 Jan 27 00:15 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1884119553 Jan 26 19:22 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1812662128 Jan 26 19:03 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   73043980 Jan 27 00:06 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242705 Jan 27 00:15 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Jan 27 00:15 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Jan 27 00:15 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Jan 27 00:15 time.db
-rw-r--r-- 1 dune01 dune01        519 Jan 26 22:25 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 26 19:22 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 27 00:15 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Jan 26 18:56 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Jan 27 00:15 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Jan 26 22:35 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Jan 26 22:25 debugprod.log
justIN time: 2026-02-04 22:16:34 UTC       justIN version: 01.06.00