justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 261204.0@dunegpschedd02.fnal.gov

Jobsub ID261204.0@dunegpschedd02.fnal.gov
Workflow ID11163
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-11 20:10:40
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2025-12-12 15:57:44
From worker nodeHostnameheplnc154.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-12 11:34:32
Input filesmonte-carlo-011163-000001
JobscriptExit code0
Real time4h (15698s)
CPU time4h (14926s = 95%)
Max RSS bytes4433506304 (4228 MiB)
Outputting started2025-12-12 15:56:11
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11163/1/001/reco_stage1.root
Finished2025-12-12 15:57:44
Saved logsjustin-logs:261204.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 170440 traces, 0 summary
Retagger: tagging trace set: gauss with 126196 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 126196 traces tagged "gauss"
FrameSaver: q=4.53778e+06 n=2181315 tag=gauss
wclsFrameSaver: saving 170440 traces tagged "wiener"
FrameSaver: q=4.81554e+06 n=1915314 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 36799 XUs and 39941 XVs -> 680 XUVs
C:0 T:1 60200 XUs and 60745 XVs -> 9010 XUVs
C:0 T:2 126772 XUs and 155347 XVs -> 6983 XUVs
C:0 T:3 64257 XUs and 62625 XVs -> 378 XUVs
C:0 T:4 15269 XUs and 13222 XVs -> 291 XUVs
C:0 T:5 209683 XUs and 115534 XVs -> 46194 XUVs
C:0 T:6 75061 XUs and 91251 XVs -> 3873 XUVs
C:0 T:7 30396 XUs and 35671 XVs -> 265 XUVs
C:0 T:8 8087 XUs and 5817 XVs -> 104 XUVs
C:0 T:9 82928 XUs and 63285 XVs -> 12146 XUVs
C:0 T:10 122048 XUs and 104228 XVs -> 17497 XUVs
C:0 T:11 355 XUs and 269 XVs -> 21 XUVs
97442 XUVs total
14189 collection wire objects
93578 potential space points
Neighbour search...
25812194 tests to find 10208330 neighbours
Iterating with no regularization...
Begin: 8.42478e+07
0 6.84348e+07
1 6.73493e+07
2 6.71879e+07
3 6.714e+07
Now with regularization...
Begin: 5.46448e+07
0 5.45832e+07
1 5.45367e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411058640
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411058640
fLifetime: 30000 [us]
12-Dec-2025 15:56:08 GMT  Closed output file "reco_stage1.root"
12-Dec-2025 15:56:08 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          384.492       535.811       630.192       534.992       77.0607        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000987504    0.0052438     0.010922     0.00433361    0.00310843       10     
reco:ophit:OpHitFinder                             0.316935      0.440949      0.519128      0.467612      0.0563733       10     
reco:opflash:OpFlashFinder                         0.193904      0.273281      0.333481      0.285259      0.0431182       10     
reco:opslicer:OpSlicer                             0.686186       1.01722       1.22561       1.08394      0.170277        10     
reco:caldata:DataPrepModule                         5.34465       7.02827       7.99901       7.41602      0.940763        10     
reco:wclsdatasp:WireCellToolkit                     38.3076       42.1443       45.2032       42.1126       2.31544        10     
reco:gaushit:GausHitFinder                         0.991482       1.78117       2.15235       1.87839      0.374877        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000568788   0.00124645    0.00269208    0.000851604   0.000746543      10     
reco:reco3d:SpacePointSolver                        13.0726       30.0597       49.6281       31.2481       10.5633        10     
reco:hitpdune:DisambigFromSpacePoints               1.5196        2.8497        4.38367         2.8        0.826388        10     
reco:pandora:StandardPandora                        40.224        59.9425       97.5837       59.569        15.0257        10     
reco:pandoraWriter:StandardPandora                 0.663039      0.818565       1.20231      0.752953      0.148706        10     
reco:pandoraTrack:LArPandoraTrackCreation           4.94219       7.49428       9.23246       7.61792       1.3815         10     
reco:pandoraShower:LArPandoraShowerCreation         4.7812        9.02004       12.6799       9.21382       2.66553        10     
reco:pandoracalo:Calorimetry                        2.00188       4.01305       4.98707       4.51672      0.988686        10     
reco:pandoracalonosce:Calorimetry                   1.91944       3.52327       4.55832       3.69836      0.840525        10     
reco:pandorapid:Chi2ParticleID                    0.00557476    0.00963723     0.0114746     0.0106117    0.00209491       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0454523     0.0903822     0.120493      0.0920085     0.0210543       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00489856    0.00940267     0.0118612    0.00983551    0.00189581       10     
reco:pandoraShowercalo:ShowerCalorimetry            3.55362       8.40259       11.8387       8.80928       2.18644        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.4322        8.22298       11.0995       8.35951       2.01645        10     
reco:emtrkmichelid:EmTrackMichelId                  217.645       269.091       304.375       273.273       29.3786        10     
reco:crttag:SingleCRTMatchingProducer              0.0168478     0.0235156     0.0335954     0.0225077    0.00525404       10     
reco:crtreco:TwoCRTMatchingProducer               0.00810483     0.0135932     0.0163453     0.0139284    0.00216866       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.875889       1.19351        1.503        1.21575      0.194705        10     
reco:pandora2Track:LArPandoraTrackCreation          11.2412       23.3808       33.1227       23.2688       6.38533        10     
reco:pandora2calo:Calorimetry                       6.48467       12.2854       16.284        12.4837       3.32115        10     
reco:pandora2calonosce:Calorimetry                  5.36588       11.3373       14.8024       12.4159       3.21952        10     
reco:pandora2pid:Chi2ParticleID                    0.0163117     0.0252266     0.028781      0.0261941    0.00351106       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0941605     0.125189      0.151076      0.122774      0.0160108       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0164152     0.0249035     0.0281142     0.0258151    0.00327744       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.38143       8.13479       11.7086       8.07626       2.47961        10     
reco:pandora2Showercalo:ShowerCalorimetry           5.88534       9.6911        14.2783       9.53615       2.58201        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      5.03014       8.78178       11.5427       8.41553       1.89554        10     
[art]:TriggerResults:TriggerResultInserter        7.0402e-05    9.88986e-05   0.000195958   8.54605e-05   3.66823e-05      10     
end_path:out1:RootOutput                           1.062e-05    5.71434e-05   0.00044164    1.39865e-05   0.000128217      10     
end_path:out1:RootOutput(write)                     3.13235       4.35943       5.25905       4.34012      0.632586        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5268.02 MB
  Peak resident set size usage (VmHWM): 4433.51 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.32 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.42 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.18 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11473148
-rw-r--r-- 1 pltdune04 pltdune 2649599385 Dec 12 14:24 detsim_stage2.root
-rw-r--r-- 1 pltdune04 pltdune 2512429343 Dec 12 14:21 detsim_stage1.root
-rw-r--r-- 1 pltdune04 pltdune 2401598310 Dec 12 15:56 reco_stage1.root
-rw-r--r-- 1 pltdune04 pltdune 2085402705 Dec 12 11:58 g4_stage2.root
-rw-r--r-- 1 pltdune04 pltdune 2008626996 Dec 12 11:42 g4_stage1.root
-rw-r--r-- 1 pltdune04 pltdune   90230160 Dec 12 15:49 Pandora_Events.pndr
-rw-r--r-- 1 pltdune04 pltdune     243297 Dec 12 15:56 jobscript.log
-rw-r--r-- 1 pltdune04 pltdune     217088 Dec 12 15:56 mem.db
-rw-r--r-- 1 pltdune04 pltdune      62376 Dec 12 15:56 Pandora_Geometry.xml
-rw-r--r-- 1 pltdune04 pltdune      36864 Dec 12 15:56 time.db
-rw-r--r-- 1 pltdune04 pltdune        519 Dec 12 14:24 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 pltdune04 pltdune        519 Dec 12 11:58 g4_protoDUNE_hist.root
-rw-r--r-- 1 pltdune04 pltdune        519 Dec 12 15:56 protoDUNE_reco1_hist.root
-rw-r--r-- 1 pltdune04 pltdune         52 Dec 12 11:34 all-input-dids.txt
-rw-r--r-- 1 pltdune04 pltdune          7 Dec 12 15:56 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune04 pltdune          0 Dec 12 14:32 ProtoDUNE.log
-rw-r--r-- 1 pltdune04 pltdune          0 Dec 12 14:24 debugprod.log
justIN time: 2026-02-07 13:31:48 UTC       justIN version: 01.06.00