justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269132.0@dunegpschedd01.fnal.gov

Jobsub ID269132.0@dunegpschedd01.fnal.gov
Workflow ID11265
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-18 22:20:15
SiteUK_Bristol
EntryCMSHTPC_T2_UK_SGrid_Bristol_lcgce02
Last heartbeat2025-12-19 05:38:24
From worker nodeHostnamehd85.dice.priv
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 22:21:09
Input filesmonte-carlo-011265-000001
JobscriptExit code0
Real time7h (26124s)
CPU time7h (25369s = 97%)
Max RSS bytes4578017280 (4365 MiB)
Outputting started2025-12-19 05:36:33
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11265/1/001/reco_stage1.root
Finished2025-12-19 05:38:24
Saved logsjustin-logs:269132.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 162360 traces, 0 summary
Retagger: tagging trace set: gauss with 122124 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 122124 traces tagged "gauss"
FrameSaver: q=4.47502e+06 n=2197270 tag=gauss
wclsFrameSaver: saving 162360 traces tagged "wiener"
FrameSaver: q=4.73703e+06 n=1945188 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 24183 XUs and 27167 XVs -> 440 XUVs
C:0 T:1 53745 XUs and 81567 XVs -> 7584 XUVs
C:0 T:2 97908 XUs and 148377 XVs -> 10019 XUVs
C:0 T:3 44581 XUs and 55056 XVs -> 409 XUVs
C:0 T:4 15979 XUs and 19523 XVs -> 217 XUVs
C:0 T:5 88066 XUs and 130668 XVs -> 11940 XUVs
C:0 T:6 171217 XUs and 179759 XVs -> 7882 XUVs
C:0 T:7 68088 XUs and 62009 XVs -> 821 XUVs
C:0 T:8 9100 XUs and 11713 XVs -> 138 XUVs
C:0 T:9 66546 XUs and 111245 XVs -> 5798 XUVs
C:0 T:10 47887 XUs and 33665 XVs -> 4284 XUVs
C:0 T:11 327 XUs and 271 XVs -> 43 XUVs
49575 XUVs total
14450 collection wire objects
45123 potential space points
Neighbour search...
4036569 tests to find 1983228 neighbours
Iterating with no regularization...
Begin: 8.96018e+07
0 6.88584e+07
1 6.73061e+07
2 6.70588e+07
3 6.69781e+07
4 6.69412e+07
Now with regularization...
Begin: 5.39956e+07
0 5.39364e+07
1 5.39055e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411200947
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411200947
fLifetime: 30000 [us]
19-Dec-2025 05:36:30 UTC  Closed output file "reco_stage1.root"
19-Dec-2025 05:36:30 UTC  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          572.721       743.656       881.975       750.065       101.241        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00195111    0.00459783    0.00617061    0.00477936    0.00137943       10     
reco:ophit:OpHitFinder                             0.393462      0.462763      0.561313      0.459976      0.0564787       10     
reco:opflash:OpFlashFinder                         0.288345      0.327648      0.394214       0.30527      0.040625        10     
reco:opslicer:OpSlicer                              1.13785       1.3565        1.77725       1.29429      0.191864        10     
reco:caldata:DataPrepModule                         7.12923       8.32519       9.66159       8.21353      0.940559        10     
reco:wclsdatasp:WireCellToolkit                     46.6802       56.0366        65.16        55.514        5.24149        10     
reco:gaushit:GausHitFinder                          1.81708       2.39588       2.96092       2.3515       0.355851        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000656055   0.000861286   0.00112712    0.000838348   0.000156057      10     
reco:reco3d:SpacePointSolver                        15.0684       29.3071       42.3525       27.2058       8.74594        10     
reco:hitpdune:DisambigFromSpacePoints               2.25172       3.36536       4.85815       3.32161      0.761384        10     
reco:pandora:StandardPandora                        58.8846       80.195        121.158       75.9724       19.6665        10     
reco:pandoraWriter:StandardPandora                  0.64206      0.794937       1.00855      0.723525      0.139814        10     
reco:pandoraTrack:LArPandoraTrackCreation           6.12544       9.29689       12.4544       9.54829       2.02156        10     
reco:pandoraShower:LArPandoraShowerCreation         7.01169       11.3076       15.5499       11.7224       2.94454        10     
reco:pandoracalo:Calorimetry                        3.24409       5.25374       7.28558       5.23991       1.25463        10     
reco:pandoracalonosce:Calorimetry                   2.98088       4.90451       6.55873       5.14772       1.19221        10     
reco:pandorapid:Chi2ParticleID                    0.00862174     0.0109931     0.0161027     0.0102128    0.00250834       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0790491     0.105256      0.139532      0.101512      0.0167362       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00695214     0.0105708     0.0154574    0.00981295    0.00256344       10     
reco:pandoraShowercalo:ShowerCalorimetry            7.19576       12.1233       16.2714       11.7089       3.17278        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       7.05999       11.4663       15.5449       11.2653       2.90513        10     
reco:emtrkmichelid:EmTrackMichelId                  321.788       397.94        452.439       418.835       48.5393        10     
reco:crttag:SingleCRTMatchingProducer              0.0264691     0.0327597     0.0487685     0.0325395    0.00583604       10     
reco:crtreco:TwoCRTMatchingProducer                0.0133846     0.022012      0.0365403     0.0208534    0.00563122       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.17934       1.69001       2.00099       1.65075      0.251559        10     
reco:pandora2Track:LArPandoraTrackCreation          19.4879       32.1114       42.2416       32.4568       7.80727        10     
reco:pandora2calo:Calorimetry                       9.47975       16.6751       26.8693       16.6255       4.61895        10     
reco:pandora2calonosce:Calorimetry                  8.31632       15.4989       20.4129       16.363        3.90855        10     
reco:pandora2pid:Chi2ParticleID                    0.0182338     0.0306053     0.0414695     0.0327256     0.0078032       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0987108     0.141025      0.173762      0.144986      0.0253483       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0174367     0.028951      0.0374336     0.0301509    0.00693817       10     
reco:pandora2Shower:LArPandoraShowerCreation        6.43075       11.436        15.2094       11.6495       2.70547        10     
reco:pandora2Showercalo:ShowerCalorimetry           7.59207       13.6051       17.2162       13.9373       3.13586        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      7.06331       12.0327       16.2273       12.3496       2.71063        10     
[art]:TriggerResults:TriggerResultInserter        7.1655e-05    9.07207e-05   0.000166463   8.27855e-05   2.63382e-05      10     
end_path:out1:RootOutput                          1.2513e-05    1.5388e-05    1.9607e-05    1.4748e-05    2.06727e-06      10     
end_path:out1:RootOutput(write)                     4.24133       5.1797        6.57004       4.97085      0.651948        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5384.44 MB
  Peak resident set size usage (VmHWM): 4578.02 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.11 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.32 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.33 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.18 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11510956
-rw-r--r-- 1 dnpil001 users 2652543375 Dec 19 03:28 detsim_stage2.root
-rw-r--r-- 1 dnpil001 users 2513318387 Dec 19 03:24 detsim_stage1.root
-rw-r--r-- 1 dnpil001 users 2383926127 Dec 19 05:36 reco_stage1.root
-rw-r--r-- 1 dnpil001 users 2115684006 Dec 18 22:57 g4_stage2.root
-rw-r--r-- 1 dnpil001 users 2035369997 Dec 18 22:32 g4_stage1.root
-rw-r--r-- 1 dnpil001 users   85760530 Dec 19 05:25 Pandora_Events.pndr
-rw-r--r-- 1 dnpil001 users     243165 Dec 19 05:36 jobscript.log
-rw-r--r-- 1 dnpil001 users     217088 Dec 19 05:36 mem.db
-rw-r--r-- 1 dnpil001 users      62376 Dec 19 05:36 Pandora_Geometry.xml
-rw-r--r-- 1 dnpil001 users      36864 Dec 19 05:36 time.db
-rw-r--r-- 1 dnpil001 users        519 Dec 19 03:28 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dnpil001 users        519 Dec 18 22:57 g4_protoDUNE_hist.root
-rw-r--r-- 1 dnpil001 users        519 Dec 19 05:36 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dnpil001 users         52 Dec 18 22:21 all-input-dids.txt
-rw-r--r-- 1 dnpil001 users          7 Dec 19 05:36 justin-processed-pfns.txt
-rw-r--r-- 1 dnpil001 users          0 Dec 19 03:38 ProtoDUNE.log
-rw-r--r-- 1 dnpil001 users          0 Dec 19 03:28 debugprod.log
justIN time: 2026-02-07 08:00:11 UTC       justIN version: 01.06.00