justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 260608.1@dunegpschedd02.fnal.gov

Jobsub ID260608.1@dunegpschedd02.fnal.gov
Workflow ID11147
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-10 23:21:30
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-12-11 03:40:39
From worker nodeHostnamewn1904250.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-10 23:23:12
Input filesmonte-carlo-011147-000001
JobscriptExit code0
Real time4h (15343s)
CPU time4h (14809s = 96%)
Max RSS bytes4385996800 (4182 MiB)
Outputting started2025-12-11 03:38:56
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11147/1/001/reco_stage1.root
Finished2025-12-11 03:40:39
Saved logsjustin-logs:260608.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 152233 traces, 0 summary
Retagger: tagging trace set: gauss with 119628 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 119628 traces tagged "gauss"
FrameSaver: q=4.35896e+06 n=2157695 tag=gauss
wclsFrameSaver: saving 152233 traces tagged "wiener"
FrameSaver: q=4.60938e+06 n=1931437 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 56311 XUs and 48895 XVs -> 684 XUVs
C:0 T:1 105073 XUs and 92957 XVs -> 5831 XUVs
C:0 T:2 190564 XUs and 208370 XVs -> 9763 XUVs
C:0 T:3 69488 XUs and 75214 XVs -> 572 XUVs
C:0 T:4 19408 XUs and 12799 XVs -> 173 XUVs
C:0 T:5 99587 XUs and 45163 XVs -> 9067 XUVs
C:0 T:6 100828 XUs and 115238 XVs -> 3600 XUVs
C:0 T:7 42283 XUs and 49920 XVs -> 498 XUVs
C:0 T:8 7502 XUs and 5851 XVs -> 76 XUVs
C:0 T:9 81099 XUs and 49743 XVs -> 7528 XUVs
C:0 T:10 52324 XUs and 26048 XVs -> 8631 XUVs
C:0 T:11 257 XUs and 175 XVs -> 38 XUVs
46461 XUVs total
13739 collection wire objects
42876 potential space points
Neighbour search...
5326794 tests to find 2482556 neighbours
Iterating with no regularization...
Begin: 1.00281e+08
0 8.27907e+07
1 8.15892e+07
2 8.1456e+07
3 8.14276e+07
Now with regularization...
Begin: 6.74627e+07
0 6.73905e+07
1 6.73531e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411041164
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411041164
fLifetime: 30000 [us]
11-Dec-2025 03:38:54 GMT  Closed output file "reco_stage1.root"
11-Dec-2025 03:38:54 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          316.696       511.186       734.842       505.226       147.485        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00146335    0.00473614     0.0110702    0.00371067    0.00283835       10     
reco:ophit:OpHitFinder                             0.248983      0.364807      0.481011      0.368893      0.081706        10     
reco:opflash:OpFlashFinder                         0.150496      0.218535      0.314292      0.224198      0.0454739       10     
reco:opslicer:OpSlicer                             0.642135      0.880975       1.20859      0.858182      0.187461        10     
reco:caldata:DataPrepModule                         5.57018       6.9071        8.1273        6.92466       0.87312        10     
reco:wclsdatasp:WireCellToolkit                     35.887        43.7998       49.7018       43.8785       4.22872        10     
reco:gaushit:GausHitFinder                          0.94855       1.67625        2.49         1.66743      0.551819        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000326293   0.000693083   0.000977699   0.000678656   0.000191451      10     
reco:reco3d:SpacePointSolver                        11.9101       25.4044       40.5223       24.0635       10.6932        10     
reco:hitpdune:DisambigFromSpacePoints               1.59201       2.6765        3.63128       2.5248       0.817668        10     
reco:pandora:StandardPandora                        23.8835       63.0816       155.227        62.1         38.8498        10     
reco:pandoraWriter:StandardPandora                 0.370898       0.60057      0.890818       0.54282      0.192299        10     
reco:pandoraTrack:LArPandoraTrackCreation           2.43064       6.32694       10.1989        6.051        2.63834        10     
reco:pandoraShower:LArPandoraShowerCreation         2.50268       7.19902       12.914        6.31862       3.59086        10     
reco:pandoracalo:Calorimetry                        1.29989       3.51698       6.35495       3.1437        1.48098        10     
reco:pandoracalonosce:Calorimetry                   1.87932       3.39074       6.35807       2.88527        1.635         10     
reco:pandorapid:Chi2ParticleID                    0.00557352     0.0085616     0.0124155    0.00863117    0.00218504       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.044453      0.0747514      0.11037      0.0736507     0.0203649       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00379442    0.00727811     0.0121001    0.00659597    0.00262211       10     
reco:pandoraShowercalo:ShowerCalorimetry            2.55576       7.56994       14.0128       6.49693       3.96676        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.20238       7.35763       13.7961       6.55799       3.98317        10     
reco:emtrkmichelid:EmTrackMichelId                  177.981       259.699       366.464       245.47        57.8774        10     
reco:crttag:SingleCRTMatchingProducer              0.012404      0.0219938     0.0437536     0.0211147    0.00837181       10     
reco:crtreco:TwoCRTMatchingProducer               0.00781204     0.0125945     0.0214465     0.0117482    0.00413409       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.769607       1.14109       1.70481       1.07266      0.267361        10     
reco:pandora2Track:LArPandoraTrackCreation          8.60595       20.8257       40.2385       16.173        10.9702        10     
reco:pandora2calo:Calorimetry                       4.64506       10.7742       21.9081       9.52174       5.32579        10     
reco:pandora2calonosce:Calorimetry                  3.53573       9.80149       19.1331       8.22129       4.96304        10     
reco:pandora2pid:Chi2ParticleID                   0.00876918     0.0222417     0.0331239     0.019135     0.00824988       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0485697     0.101756      0.156639      0.0986432     0.0332424       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00836468     0.0193542     0.0309912     0.0168596    0.00760792       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.62008       7.14077       13.957        6.54586       3.32232        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.77963       8.92739       17.8582       8.21011       4.33515        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.57045       7.66317       14.5026       6.04657       3.72716        10     
[art]:TriggerResults:TriggerResultInserter        4.5706e-05    7.8661e-05    0.000196199   7.2732e-05    4.08891e-05      10     
end_path:out1:RootOutput                           9.417e-06    1.25357e-05   2.3254e-05    1.1056e-05    3.80758e-06      10     
end_path:out1:RootOutput(write)                     2.82084       3.81646       5.34486       3.44813      0.882454        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5204.45 MB
  Peak resident set size usage (VmHWM): 4386 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.10 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.31 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.35 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.18 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10978680
-rw-r--r-- 1 dune01 dune01 2562461666 Dec 11 02:11 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2430835574 Dec 11 02:09 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2289567006 Dec 11 03:38 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1983000565 Dec 10 23:46 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1891038786 Dec 10 23:30 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   84637210 Dec 11 03:31 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242851 Dec 11 03:38 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Dec 11 03:38 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Dec 11 03:38 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Dec 11 03:38 time.db
-rw-r--r-- 1 dune01 dune01        519 Dec 11 02:11 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec 10 23:46 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec 11 03:38 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Dec 10 23:23 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Dec 11 03:38 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Dec 11 02:16 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Dec 11 02:11 debugprod.log
justIN time: 2026-02-07 13:30:01 UTC       justIN version: 01.06.00