justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 296620.0@dunegpschedd02.fnal.gov

Jobsub ID296620.0@dunegpschedd02.fnal.gov
Workflow ID12548
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-30 07:26:55
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-01-30 11:39:00
From worker nodeHostnamewn2208251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-30 07:29:17
Input filesmonte-carlo-012548-000001
JobscriptExit code0
Real time4h (14882s)
CPU time4h (14467s = 97%)
Max RSS bytes4148523008 (3956 MiB)
Outputting started2026-01-30 11:37:20
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12548/1/001/reco_stage1.root
Finished2026-01-30 11:39:00
Saved logsjustin-logs:296620.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 137324 traces, 0 summary
Retagger: tagging trace set: gauss with 110205 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 110205 traces tagged "gauss"
FrameSaver: q=3.71215e+06 n=2034981 tag=gauss
wclsFrameSaver: saving 137324 traces tagged "wiener"
FrameSaver: q=3.89687e+06 n=1822567 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 40397 XUs and 42872 XVs -> 424 XUVs
C:0 T:1 67781 XUs and 61391 XVs -> 4399 XUVs
C:0 T:2 106934 XUs and 123613 XVs -> 4641 XUVs
C:0 T:3 49592 XUs and 48058 XVs -> 233 XUVs
C:0 T:4 15499 XUs and 10716 XVs -> 77 XUVs
C:0 T:5 72569 XUs and 42173 XVs -> 5514 XUVs
C:0 T:6 153762 XUs and 145865 XVs -> 5777 XUVs
C:0 T:7 55862 XUs and 51652 XVs -> 635 XUVs
C:0 T:8 9067 XUs and 7353 XVs -> 135 XUVs
C:0 T:9 47377 XUs and 49626 XVs -> 2873 XUVs
C:0 T:10 35990 XUs and 25988 XVs -> 3036 XUVs
C:0 T:11 243 XUs and 213 XVs -> 32 XUVs
27776 XUVs total
13263 collection wire objects
25243 potential space points
Neighbour search...
1088533 tests to find 537988 neighbours
Iterating with no regularization...
Begin: 8.14054e+07
0 7.07323e+07
1 6.9972e+07
2 6.98755e+07
3 6.98523e+07
Now with regularization...
Begin: 5.86159e+07
0 5.85947e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 412053783
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 412053783
fLifetime: 30000 [us]
30-Jan-2026 11:37:17 GMT  Closed output file "reco_stage1.root"
30-Jan-2026 11:37:17 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          279.096       415.265       543.445       416.424       92.8433        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00127931    0.00349497    0.00768045    0.00314127    0.00167507       10     
reco:ophit:OpHitFinder                             0.249318       0.32978      0.416254      0.324018      0.0578037       10     
reco:opflash:OpFlashFinder                          0.1232       0.179799       0.22945      0.179581      0.028202        10     
reco:opslicer:OpSlicer                             0.479835      0.798799       1.06835      0.781181      0.159288        10     
reco:caldata:DataPrepModule                         5.25498       6.86469       7.86301       7.2242       0.900712        10     
reco:wclsdatasp:WireCellToolkit                     34.293        41.4351       47.2795       43.2843       4.88038        10     
reco:gaushit:GausHitFinder                          1.09109       1.58006       2.09636       1.57697      0.293738        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000475824   0.000677651   0.000919497   0.000678721   0.000124895      10     
reco:reco3d:SpacePointSolver                        9.3035        18.3187       26.5669       18.0526       6.06407        10     
reco:hitpdune:DisambigFromSpacePoints               1.09446       2.0147        2.8991        2.08164      0.594987        10     
reco:pandora:StandardPandora                        18.1668       39.2965       59.0959       34.7327       13.9375        10     
reco:pandoraWriter:StandardPandora                 0.317235      0.544475      0.760221      0.560777       0.12981        10     
reco:pandoraTrack:LArPandoraTrackCreation           2.06655       3.88842       5.60723       3.96085       1.11334        10     
reco:pandoraShower:LArPandoraShowerCreation         1.96111       4.74481       8.37401       4.78703       1.92585        10     
reco:pandoracalo:Calorimetry                        1.18289       2.47454       3.34488       2.7236       0.717924        10     
reco:pandoracalonosce:Calorimetry                   1.05068       2.20294       2.88069       2.47563      0.650985        10     
reco:pandorapid:Chi2ParticleID                     0.0045753    0.00661469    0.00872234     0.005979      0.0015436       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0395512     0.0654125     0.0834792     0.0647223     0.0153616       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00341262    0.00664775     0.0090422    0.00659704    0.00176597       10     
reco:pandoraShowercalo:ShowerCalorimetry            2.05961       5.17157       9.06574       5.3275        2.1458         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.96449       5.11116       8.39117       5.33181       2.01805        10     
reco:emtrkmichelid:EmTrackMichelId                  164.52        229.864       294.245       242.229       48.0685        10     
reco:crttag:SingleCRTMatchingProducer              0.014773       0.01944      0.0286687     0.0178517    0.00439553       10     
reco:crtreco:TwoCRTMatchingProducer               0.00768413     0.0108873     0.0141211     0.0110056    0.00159759       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.745853       1.08235       1.37408       1.10602      0.182984        10     
reco:pandora2Track:LArPandoraTrackCreation           5.788        14.382        25.2566        13.4         5.87922        10     
reco:pandora2calo:Calorimetry                       2.62935       7.19411        12.89        6.58731       3.17195        10     
reco:pandora2calonosce:Calorimetry                  3.02164       6.89731       12.2797       5.99188       2.89541        10     
reco:pandora2pid:Chi2ParticleID                    0.0100919     0.0185888     0.0270546     0.018645     0.00592064       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0517617     0.0923638     0.131655      0.0968882     0.0255143       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0101153     0.0182602     0.0263303     0.0179559    0.00528068       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.40729       5.06777       8.50016       4.6393        1.88982        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.77932       6.30734       10.9376       5.76936       2.54818        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.67067       5.76915       9.46106       5.45464       2.18361        10     
[art]:TriggerResults:TriggerResultInserter        5.6376e-05    8.94792e-05   0.000210615   8.1178e-05    4.23923e-05      10     
end_path:out1:RootOutput                           9.899e-06    1.29333e-05   2.2312e-05    1.24885e-05   3.44323e-06      10     
end_path:out1:RootOutput(write)                     2.1688        3.38101       4.49144       3.37338      0.710161        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4979.33 MB
  Peak resident set size usage (VmHWM): 4142.28 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.31 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.24 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10045944
-rw-r--r-- 1 dune01 dune01 2358072469 Jan 30 10:25 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2240250320 Jan 30 10:23 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2045591705 Jan 30 11:37 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1812083983 Jan 30 07:50 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1755217866 Jan 30 07:36 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   75195580 Jan 30 11:30 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242565 Jan 30 11:37 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Jan 30 11:37 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Jan 30 11:37 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Jan 30 11:37 time.db
-rw-r--r-- 1 dune01 dune01        519 Jan 30 10:26 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 30 07:50 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 30 11:37 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Jan 30 07:29 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Jan 30 11:37 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Jan 30 10:31 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Jan 30 10:26 debugprod.log
justIN time: 2026-02-04 13:58:51 UTC       justIN version: 01.06.00