justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 295946.1@dunegpschedd02.fnal.gov

Jobsub ID295946.1@dunegpschedd02.fnal.gov
Workflow ID12446
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-27 14:23:30
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-01-27 18:05:14
From worker nodeHostnamewn1207250.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-27 14:25:17
Input filesmonte-carlo-012446-000001
JobscriptExit code0
Real time3h (13096s)
CPU time3h (12193s = 93%)
Max RSS bytes4052426752 (3864 MiB)
Outputting started2026-01-27 18:03:34
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12446/1/001/reco_stage1.root
Finished2026-01-27 18:05:14
Saved logsjustin-logs:295946.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

KeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 111119 traces, 0 summary
Retagger: tagging trace set: gauss with 84631 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 84631 traces tagged "gauss"
FrameSaver: q=2.91017e+06 n=1507156 tag=gauss
wclsFrameSaver: saving 111119 traces tagged "wiener"
FrameSaver: q=3.09324e+06 n=1336572 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 33081 XUs and 32888 XVs -> 264 XUVs
C:0 T:1 62654 XUs and 59925 XVs -> 2921 XUVs
C:0 T:2 61046 XUs and 56794 XVs -> 3523 XUVs
C:0 T:3 29266 XUs and 21480 XVs -> 164 XUVs
C:0 T:4 15324 XUs and 14539 XVs -> 305 XUVs
C:0 T:5 91575 XUs and 61621 XVs -> 24803 XUVs
C:0 T:6 79488 XUs and 80358 XVs -> 4523 XUVs
C:0 T:7 31456 XUs and 29857 XVs -> 189 XUVs
C:0 T:8 7993 XUs and 6743 XVs -> 74 XUVs
C:0 T:9 48065 XUs and 53740 XVs -> 5340 XUVs
C:0 T:10 22186 XUs and 17886 XVs -> 2814 XUVs
C:0 T:11 48 XUs and 27 XVs -> 7 XUVs
44927 XUVs total
9708 collection wire objects
42797 potential space points
Neighbour search...
9640825 tests to find 4032788 neighbours
Iterating with no regularization...
Begin: 5.679e+07
0 4.74445e+07
1 4.68779e+07
2 4.68024e+07
3 4.67815e+07
Now with regularization...
Begin: 3.91587e+07
0 3.91088e+07
1 3.90756e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411999264
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411999264
fLifetime: 30000 [us]
27-Jan-2026 18:03:31 GMT  Closed output file "reco_stage1.root"
27-Jan-2026 18:03:31 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          324.445       587.702       806.245       572.641       152.413        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000810609   0.00420799    0.00665287    0.00499818    0.00177516       10     
reco:ophit:OpHitFinder                              0.29071      0.429121      0.531498      0.431188      0.0717065       10     
reco:opflash:OpFlashFinder                         0.158789      0.236335      0.307005       0.22445      0.0524962       10     
reco:opslicer:OpSlicer                             0.621386       1.1741        1.6597        1.1669       0.282418        10     
reco:caldata:DataPrepModule                         5.43122       8.4896        9.27184       8.84596       1.04876        10     
reco:wclsdatasp:WireCellToolkit                     33.056        54.3173       60.6616       58.7425       8.25479        10     
reco:gaushit:GausHitFinder                         0.904994       1.8679        2.48021       1.94396       0.4739         10     
reco:nhitsfilter:NumberOfHitsFilter               0.00055637    0.000777345   0.000968909   0.00081839    0.000146484      10     
reco:reco3d:SpacePointSolver                        9.20226       21.4057       37.1422       21.472        8.68356        10     
reco:hitpdune:DisambigFromSpacePoints              0.938096       2.43147       3.58851       2.30822      0.868187        10     
reco:pandora:StandardPandora                        17.5579       48.4384       105.25        42.4711       24.6315        10     
reco:pandoraWriter:StandardPandora                 0.313949      0.645259      0.927089      0.588437       0.17244        10     
reco:pandoraTrack:LArPandoraTrackCreation           2.83263       6.06063       9.26313       6.11372       2.14097        10     
reco:pandoraShower:LArPandoraShowerCreation         2.13308       6.9401        13.4108       6.19539       3.59841        10     
reco:pandoracalo:Calorimetry                        1.39888       3.39654       5.0936        3.47326       1.26421        10     
reco:pandoracalonosce:Calorimetry                   1.2763        3.20858       4.74425       3.25417       1.19705        10     
reco:pandorapid:Chi2ParticleID                    0.00701565    0.00968715     0.0120373    0.00993735    0.00161944       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0418977     0.0915373     0.124002      0.0891088     0.0260919       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00546827    0.00916855     0.0107997    0.00984396    0.00172796       10     
reco:pandoraShowercalo:ShowerCalorimetry            2.00041       7.11999       13.1871       6.6476        3.6772         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.93199       7.15687       12.7594       7.00901       3.51847        10     
reco:emtrkmichelid:EmTrackMichelId                  200.361       343.014       451.175       336.339       74.5104        10     
reco:crttag:SingleCRTMatchingProducer              0.0123687     0.0216313     0.0308839     0.021054     0.00540581       10     
reco:crtreco:TwoCRTMatchingProducer                0.0106488     0.0157743     0.0227674     0.014778     0.00434116       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.04204       1.39018       1.84129       1.34937      0.244998        10     
reco:pandora2Track:LArPandoraTrackCreation          11.0225       21.1798       34.1619       20.0542       8.06384        10     
reco:pandora2calo:Calorimetry                       5.01423       10.2427       17.253        9.86641       4.31301        10     
reco:pandora2calonosce:Calorimetry                  4.9208        9.56238       16.2356       9.17067       3.70117        10     
reco:pandora2pid:Chi2ParticleID                    0.0184145     0.0250726     0.0321939     0.0241376    0.00411146       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.100067       0.12499      0.167737      0.114287      0.0235658       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0186752     0.0246532     0.0319484     0.0239718    0.00388235       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.00303       7.8847        12.7514       7.40892       2.95467        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.73269       8.63394       14.4184       7.52475       3.64738        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.45616       7.3869        12.7784       6.33597       2.95504        10     
[art]:TriggerResults:TriggerResultInserter        3.4465e-05    6.48844e-05   0.00014854    5.2439e-05    3.21148e-05      10     
end_path:out1:RootOutput                           6.342e-06    1.07984e-05   4.1479e-05     7.469e-06    1.02365e-05      10     
end_path:out1:RootOutput(write)                     3.68962       4.61285       5.99079       4.45997      0.748883        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4890.51 MB
  Peak resident set size usage (VmHWM): 4052.43 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.35 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.50 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.22 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9531856
-rw-r--r-- 1 dune01 dune01 2256005507 Jan 27 16:22 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2140345878 Jan 27 16:20 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1918102147 Jan 27 18:03 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1724795986 Jan 27 14:48 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1651679129 Jan 27 14:36 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   69065060 Jan 27 17:56 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242782 Jan 27 18:03 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Jan 27 18:03 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Jan 27 18:03 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Jan 27 18:03 time.db
-rw-r--r-- 1 dune01 dune01        519 Jan 27 16:22 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 27 14:48 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 27 18:03 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Jan 27 14:25 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Jan 27 18:03 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Jan 27 16:27 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Jan 27 16:22 debugprod.log
justIN time: 2026-02-04 18:08:27 UTC       justIN version: 01.06.00