justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 295857.0@dunegpschedd02.fnal.gov

Jobsub ID295857.0@dunegpschedd02.fnal.gov
Workflow ID12424
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-27 08:23:09
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-01-27 11:31:03
From worker nodeHostnamewn1207091.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-27 08:24:13
Input filesmonte-carlo-012424-000001
JobscriptExit code0
Real time3h (11121s)
CPU time2h (10774s = 96%)
Max RSS bytes4212240384 (4017 MiB)
Outputting started2026-01-27 11:29:35
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12424/1/001/reco_stage1.root
Finished2026-01-27 11:31:03
Saved logsjustin-logs:295857.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ol adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 132043 traces, 0 summary
Retagger: tagging trace set: gauss with 101841 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 101841 traces tagged "gauss"
FrameSaver: q=3.40682e+06 n=1802555 tag=gauss
wclsFrameSaver: saving 132043 traces tagged "wiener"
FrameSaver: q=3.58971e+06 n=1597012 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 33640 XUs and 32671 XVs -> 376 XUVs
C:0 T:1 73194 XUs and 67301 XVs -> 5953 XUVs
C:0 T:2 68345 XUs and 97781 XVs -> 4422 XUVs
C:0 T:3 35796 XUs and 32922 XVs -> 203 XUVs
C:0 T:4 19850 XUs and 18375 XVs -> 237 XUVs
C:0 T:5 85007 XUs and 62081 XVs -> 5425 XUVs
C:0 T:6 39877 XUs and 69562 XVs -> 1970 XUVs
C:0 T:7 16655 XUs and 26210 XVs -> 96 XUVs
C:0 T:8 11712 XUs and 10074 XVs -> 260 XUVs
C:0 T:9 77907 XUs and 78440 XVs -> 9075 XUVs
C:0 T:10 26689 XUs and 57736 XVs -> 6855 XUVs
C:0 T:11 163 XUs and 93 XVs -> 21 XUVs
34893 XUVs total
11229 collection wire objects
32740 potential space points
Neighbour search...
3957540 tests to find 1829440 neighbours
Iterating with no regularization...
Begin: 6.97887e+07
0 5.58867e+07
1 5.52534e+07
2 5.51769e+07
3 5.51545e+07
Now with regularization...
Begin: 4.61633e+07
0 4.61178e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411994279
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411994279
fLifetime: 30000 [us]
27-Jan-2026 11:29:33 GMT  Closed output file "reco_stage1.root"
27-Jan-2026 11:29:33 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          247.176       335.273       469.366       324.282       69.6604        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000856697   0.00306585    0.00521254    0.00325018    0.00132917       10     
reco:ophit:OpHitFinder                             0.213429      0.261746      0.301554      0.266225      0.0263509       10     
reco:opflash:OpFlashFinder                         0.108036      0.146802      0.191612      0.151645      0.0240124       10     
reco:opslicer:OpSlicer                             0.454726      0.644948      0.749358      0.686701      0.0850496       10     
reco:caldata:DataPrepModule                         4.80893       5.03013       5.44743       5.01292      0.162196        10     
reco:wclsdatasp:WireCellToolkit                     31.5071       35.4217       37.6163       36.5264       1.94608        10     
reco:gaushit:GausHitFinder                         0.743887       1.0598        1.46022       1.04268      0.235738        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000386179   0.000536611   0.000709159   0.000541527   9.26992e-05      10     
reco:reco3d:SpacePointSolver                        7.9986        18.327        53.0554       14.4657       13.2929        10     
reco:hitpdune:DisambigFromSpacePoints              0.800241       1.48255       2.39691       1.4661       0.449825        10     
reco:pandora:StandardPandora                        13.9646       30.6677       48.7979       30.4487       12.4835        10     
reco:pandoraWriter:StandardPandora                 0.278813      0.394583      0.542535      0.394438      0.0763171       10     
reco:pandoraTrack:LArPandoraTrackCreation           1.93295       3.27105       5.25963       3.05153       1.1319         10     
reco:pandoraShower:LArPandoraShowerCreation         1.54192       3.71191       8.31877       3.29287        1.931         10     
reco:pandoracalo:Calorimetry                        1.01562       1.72212       2.72077       1.65211      0.568283        10     
reco:pandoracalonosce:Calorimetry                  0.902269       1.54952       2.43883       1.51403      0.516401        10     
reco:pandorapid:Chi2ParticleID                    0.00392672    0.00512144    0.00607122    0.00502438    0.000756319      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0320614     0.0467221     0.0686176     0.044366      0.0100516       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00396396    0.00488386    0.00591862    0.00482967    0.000666408      10     
reco:pandoraShowercalo:ShowerCalorimetry            1.44586       3.53236       7.52125       3.24382       1.75683        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.40652       3.41243       6.89955       3.24185       1.61601        10     
reco:emtrkmichelid:EmTrackMichelId                  132.755       187.496       257.639       186.063       35.4952        10     
reco:crttag:SingleCRTMatchingProducer             0.00574767    0.00935142     0.0136383    0.00931143    0.00254176       10     
reco:crtreco:TwoCRTMatchingProducer               0.00492221    0.00786479     0.011857     0.00785249    0.00195013       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.548419      0.818025       1.14693      0.802367      0.179278        10     
reco:pandora2Track:LArPandoraTrackCreation          5.17071       11.0293       22.1538       9.94739       4.93337        10     
reco:pandora2calo:Calorimetry                       2.35297       5.00983       9.51279       4.79357       2.09067        10     
reco:pandora2calonosce:Calorimetry                  2.22927       4.75923       9.17721       4.49469       2.02174        10     
reco:pandora2pid:Chi2ParticleID                   0.00984487     0.0127029     0.0169964     0.012562      0.0023169       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0426008     0.0597832     0.086407       0.05695      0.0124743       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00975364     0.0128221     0.0175114     0.0125892    0.00238903       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.00089       4.08556       8.13164       3.70337       1.75926        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.06214       4.28206       8.04323       4.17469       1.76578        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      1.91681       3.96027       7.23124       3.81347       1.58405        10     
[art]:TriggerResults:TriggerResultInserter        2.4897e-05    3.3273e-05    9.2254e-05    2.7251e-05    1.96964e-05      10     
end_path:out1:RootOutput                           4.789e-06    5.7517e-06     9.688e-06     5.415e-06    1.34453e-06      10     
end_path:out1:RootOutput(write)                     2.06567       2.92161       4.1299        2.90702      0.581425        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5035.25 MB
  Peak resident set size usage (VmHWM): 4212.24 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.20 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.16 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.12 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9674360
-rw-r--r-- 1 dune01 dune01 2280366556 Jan 27 10:31 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2170700745 Jan 27 10:29 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1954674926 Jan 27 11:29 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1746519199 Jan 27 08:41 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1681649412 Jan 27 08:30 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   72002610 Jan 27 11:25 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     243005 Jan 27 11:29 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Jan 27 11:29 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Jan 27 11:29 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Jan 27 11:29 time.db
-rw-r--r-- 1 dune01 dune01        519 Jan 27 10:31 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 27 08:41 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 27 11:29 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Jan 27 08:24 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Jan 27 11:29 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Jan 27 10:36 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Jan 27 10:31 debugprod.log
justIN time: 2026-02-04 19:28:29 UTC       justIN version: 01.06.00