justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 271138.1@dunegpschedd01.fnal.gov

Jobsub ID271138.1@dunegpschedd01.fnal.gov
Workflow ID11450
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-31 23:20:29
SiteUK_Bristol
EntryCMSHTPC_T2_UK_SGrid_Bristol_lcgce02
Last heartbeat2026-01-01 04:05:15
From worker nodeHostnamehd83.dice.priv
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-31 23:21:32
Input filesmonte-carlo-011450-000001
JobscriptExit code0
Real time4h (16938s)
CPU time4h (16512s = 97%)
Max RSS bytes3956920320 (3773 MiB)
Outputting started2026-01-01 04:03:51
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11450/1/001/reco_stage1.root
Finished2026-01-01 04:05:15
Saved logsjustin-logs:271138.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ervice:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 124682 traces, 0 summary
Retagger: tagging trace set: gauss with 93422 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 93422 traces tagged "gauss"
FrameSaver: q=3.17568e+06 n=1615651 tag=gauss
wclsFrameSaver: saving 124682 traces tagged "wiener"
FrameSaver: q=3.36074e+06 n=1413869 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 44599 XUs and 39917 XVs -> 871 XUVs
C:0 T:1 127175 XUs and 148112 XVs -> 29898 XUVs
C:0 T:2 77762 XUs and 130265 XVs -> 5365 XUVs
C:0 T:3 37217 XUs and 51362 XVs -> 1220 XUVs
C:0 T:4 13944 XUs and 15515 XVs -> 170 XUVs
C:0 T:5 77511 XUs and 89198 XVs -> 15653 XUVs
C:0 T:6 58276 XUs and 74601 XVs -> 3677 XUVs
C:0 T:7 24755 XUs and 33122 XVs -> 249 XUVs
C:0 T:8 3141 XUs and 4385 XVs -> 165 XUVs
C:0 T:9 64182 XUs and 125441 XVs -> 24243 XUVs
C:0 T:10 30710 XUs and 13087 XVs -> 3179 XUVs
C:0 T:11 66 XUs and 41 XVs -> 5 XUVs
84695 XUVs total
10507 collection wire objects
79339 potential space points
Neighbour search...
25740359 tests to find 9958040 neighbours
Iterating with no regularization...
Begin: 5.83902e+07
0 4.84554e+07
1 4.77903e+07
2 4.76929e+07
3 4.76646e+07
Now with regularization...
Begin: 3.87223e+07
0 3.86769e+07
1 3.86362e+07
2 3.8601e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411463636
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411463636
fLifetime: 30000 [us]
01-Jan-2026 04:03:48 UTC  Closed output file "reco_stage1.root"
01-Jan-2026 04:03:48 UTC  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          289.236       435.034       603.558       444.168        85.68         10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00117093    0.00300618     0.0042954    0.00304035    0.000943202      10     
reco:ophit:OpHitFinder                             0.259804      0.321612      0.372918      0.331353       0.03328        10     
reco:opflash:OpFlashFinder                         0.119446      0.194918      0.238268      0.206401      0.0339932       10     
reco:opslicer:OpSlicer                             0.660627      0.893353       1.0788        0.88835      0.124664        10     
reco:caldata:DataPrepModule                         6.7104        7.01973       7.24568       7.03239      0.160937        10     
reco:wclsdatasp:WireCellToolkit                     41.818        44.9092       48.5157       44.858        1.92654        10     
reco:gaushit:GausHitFinder                          1.1304        1.50418       2.0058        1.5523        0.24712        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000408581   0.000572153   0.000792004   0.000561569   9.80059e-05      10     
reco:reco3d:SpacePointSolver                        6.60954       18.2634       28.5932       19.3865       7.33762        10     
reco:hitpdune:DisambigFromSpacePoints               1.06481       1.88439       2.92703       1.88818      0.510074        10     
reco:pandora:StandardPandora                        19.8519       35.9436       71.4432       34.9376       14.8063        10     
reco:pandoraWriter:StandardPandora                 0.353892      0.506967      0.680513      0.512084      0.0875337       10     
reco:pandoraTrack:LArPandoraTrackCreation           2.6856        5.08647       7.31063       5.33452       1.37292        10     
reco:pandoraShower:LArPandoraShowerCreation         2.13296       5.14226       9.02305       4.87894       1.90082        10     
reco:pandoracalo:Calorimetry                        1.34922       2.68388       4.00186       2.76596      0.763902        10     
reco:pandoracalonosce:Calorimetry                   1.19534       2.45963       3.70362       2.52952      0.746523        10     
reco:pandorapid:Chi2ParticleID                    0.00613055    0.00759983     0.009234     0.00742019    0.000993114      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0507266     0.0745802     0.0889112     0.0773219     0.0111356       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00586338    0.00719286    0.00851582    0.00725624    0.000783869      10     
reco:pandoraShowercalo:ShowerCalorimetry            1.94101       5.21375       9.5701        5.01168       2.23137        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.91911       5.1074        9.09396       5.44092       1.98663        10     
reco:emtrkmichelid:EmTrackMichelId                  169.87        244.765       321.571       251.063       39.7383        10     
reco:crttag:SingleCRTMatchingProducer              0.0104339     0.0172953     0.025563      0.0180293    0.00418681       10     
reco:crtreco:TwoCRTMatchingProducer               0.00715769     0.0123135     0.0184661     0.0125153    0.00294798       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.682924       1.17818       1.5105        1.20982      0.242214        10     
reco:pandora2Track:LArPandoraTrackCreation          7.16089       15.6345       25.7827       15.474        5.11892        10     
reco:pandora2calo:Calorimetry                       3.34745       7.4484        12.3843       7.39822       2.56564        10     
reco:pandora2calonosce:Calorimetry                  3.09826       7.15316        12.11        7.1058        2.56536        10     
reco:pandora2pid:Chi2ParticleID                    0.0137647     0.0181921     0.0240318     0.0181916    0.00287787       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0641489     0.0932588     0.121782      0.094301      0.0153785       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0137271     0.0179589      0.02328      0.0177515    0.00275232       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.72799       5.89192       9.9457        5.82044       1.93251        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.86162       6.3144        10.7049       6.35158       2.16221        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.6375        5.67551       9.40651       5.6477        1.82677        10     
[art]:TriggerResults:TriggerResultInserter        5.9092e-05    7.68431e-05   0.000170532   6.50375e-05   3.20391e-05      10     
end_path:out1:RootOutput                          1.2855e-05    1.42701e-05   1.8615e-05    1.3786e-05    1.60259e-06      10     
end_path:out1:RootOutput(write)                     2.25106       3.46385       4.5382        3.59402      0.618656        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4771.41 MB
  Peak resident set size usage (VmHWM): 3956.92 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.27 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.57 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.14 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9524964
-rw-r--r-- 1 dnpil001 users 2254779751 Jan  1 02:48 detsim_stage2.root
-rw-r--r-- 1 dnpil001 users 2144655255 Jan  1 02:45 detsim_stage1.root
-rw-r--r-- 1 dnpil001 users 1908987347 Jan  1 04:03 reco_stage1.root
-rw-r--r-- 1 dnpil001 users 1722311122 Dec 31 23:44 g4_stage2.root
-rw-r--r-- 1 dnpil001 users 1655461790 Dec 31 23:28 g4_stage1.root
-rw-r--r-- 1 dnpil001 users   66750220 Jan  1 03:58 Pandora_Events.pndr
-rw-r--r-- 1 dnpil001 users     242832 Jan  1 04:03 jobscript.log
-rw-r--r-- 1 dnpil001 users     217088 Jan  1 04:03 mem.db
-rw-r--r-- 1 dnpil001 users      62376 Jan  1 04:03 Pandora_Geometry.xml
-rw-r--r-- 1 dnpil001 users      36864 Jan  1 04:03 time.db
-rw-r--r-- 1 dnpil001 users        519 Jan  1 02:48 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dnpil001 users        519 Dec 31 23:44 g4_protoDUNE_hist.root
-rw-r--r-- 1 dnpil001 users        519 Jan  1 04:03 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dnpil001 users         52 Dec 31 23:21 all-input-dids.txt
-rw-r--r-- 1 dnpil001 users          7 Jan  1 04:03 justin-processed-pfns.txt
-rw-r--r-- 1 dnpil001 users          0 Jan  1 02:55 ProtoDUNE.log
-rw-r--r-- 1 dnpil001 users          0 Jan  1 02:48 debugprod.log
justIN time: 2026-02-06 23:33:08 UTC       justIN version: 01.06.00