justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 296472.0@dunegpschedd02.fnal.gov

Jobsub ID296472.0@dunegpschedd02.fnal.gov
Workflow ID12509
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-29 09:33:46
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-01-29 14:42:55
From worker nodeHostnamewn1205241.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-29 09:35:59
Input filesmonte-carlo-012509-000001
JobscriptExit code0
Real time5h (18316s)
CPU time4h (17738s = 96%)
Max RSS bytes4360019968 (4158 MiB)
Outputting started2026-01-29 14:41:17
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12509/1/001/reco_stage1.root
Finished2026-01-29 14:42:55
Saved logsjustin-logs:296472.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 141969 traces, 0 summary
Retagger: tagging trace set: gauss with 104035 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 104035 traces tagged "gauss"
FrameSaver: q=3.56519e+06 n=1751642 tag=gauss
wclsFrameSaver: saving 141969 traces tagged "wiener"
FrameSaver: q=3.79371e+06 n=1519812 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 30771 XUs and 33558 XVs -> 387 XUVs
C:0 T:1 95829 XUs and 141030 XVs -> 18095 XUVs
C:0 T:2 57418 XUs and 64935 XVs -> 4239 XUVs
C:0 T:3 28231 XUs and 30888 XVs -> 269 XUVs
C:0 T:4 13008 XUs and 13705 XVs -> 106 XUVs
C:0 T:5 62210 XUs and 69710 XVs -> 8402 XUVs
C:0 T:6 145949 XUs and 133843 XVs -> 9728 XUVs
C:0 T:7 56804 XUs and 50024 XVs -> 1057 XUVs
C:0 T:8 9362 XUs and 8138 XVs -> 80 XUVs
C:0 T:9 63430 XUs and 70819 XVs -> 4070 XUVs
C:0 T:10 23637 XUs and 21938 XVs -> 4514 XUVs
C:0 T:11 80 XUs and 91 XVs -> 24 XUVs
50971 XUVs total
12393 collection wire objects
45092 potential space points
Neighbour search...
6954254 tests to find 3085324 neighbours
Iterating with no regularization...
Begin: 7.37344e+07
0 6.25113e+07
1 6.167e+07
2 6.15484e+07
3 6.15098e+07
Now with regularization...
Begin: 5.01853e+07
0 5.0162e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 412035412
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 412035412
fLifetime: 30000 [us]
29-Jan-2026 14:41:14 GMT  Closed output file "reco_stage1.root"
29-Jan-2026 14:41:14 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          324.826       550.653       846.39        551.41        139.166        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00166735     0.0103374     0.0638388    0.00444437     0.0178853       10     
reco:ophit:OpHitFinder                             0.289767       0.39838      0.512972       0.41536      0.0652493       10     
reco:opflash:OpFlashFinder                         0.151894      0.224546      0.333297      0.234606      0.0526031       10     
reco:opslicer:OpSlicer                             0.716883       1.05796       1.58696       1.06397      0.282862        10     
reco:caldata:DataPrepModule                         5.86404       7.78359       9.47425       7.89034       1.04216        10     
reco:wclsdatasp:WireCellToolkit                      47.27        51.3384       56.8917       51.6372       3.01532        10     
reco:gaushit:GausHitFinder                         0.741134       1.54579       2.39188       1.48837      0.502552        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000390845   0.000787102   0.00111912    0.000777343   0.000219295      10     
reco:reco3d:SpacePointSolver                        5.72266       20.0084       32.1107       20.6698       8.02457        10     
reco:hitpdune:DisambigFromSpacePoints               1.04816       2.16497       4.11364       2.00983      0.906399        10     
reco:pandora:StandardPandora                        17.0771       47.1269       97.7561       41.3269       23.0547        10     
reco:pandoraWriter:StandardPandora                 0.286437      0.614097      0.897472      0.587249      0.190559        10     
reco:pandoraTrack:LArPandoraTrackCreation           1.88517       6.26154        10.3         6.30376       2.67563        10     
reco:pandoraShower:LArPandoraShowerCreation         1.7357        7.27493       11.7941       7.80266       3.1321         10     
reco:pandoracalo:Calorimetry                       0.852003       3.48773       5.91389       3.50442       1.53153        10     
reco:pandoracalonosce:Calorimetry                  0.744422       3.2369        5.74263       3.20989       1.43361        10     
reco:pandorapid:Chi2ParticleID                    0.00411218    0.00823252     0.0131425    0.00788071    0.00242275       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0331478     0.0953959     0.165716      0.0863387     0.0395189       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.004036     0.00816506     0.0107101    0.00879575    0.00207459       10     
reco:pandoraShowercalo:ShowerCalorimetry            1.65338       7.94361       14.5536       7.83332       3.88627        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.70182       7.38309       13.9646       6.7774        3.60863        10     
reco:emtrkmichelid:EmTrackMichelId                  205.843       309.501       453.589       282.179       66.8972        10     
reco:crttag:SingleCRTMatchingProducer              0.010106      0.0182895     0.0329054     0.0161422    0.00718151       10     
reco:crtreco:TwoCRTMatchingProducer               0.00850906     0.0155487     0.0287131     0.0146176    0.00591504       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.854536       1.24132       2.03467       1.15579      0.335346        10     
reco:pandora2Track:LArPandoraTrackCreation          7.7441        21.7021        40.17        21.7137       8.74972        10     
reco:pandora2calo:Calorimetry                       3.75198       10.5511       21.5052       9.88471       5.18475        10     
reco:pandora2calonosce:Calorimetry                  3.80359       10.7956       21.4542       10.0879       5.52655        10     
reco:pandora2pid:Chi2ParticleID                    0.0127706     0.022618      0.0306092     0.0232991    0.00568484       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0651641      0.11817      0.186469      0.122898      0.0378836       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0129683     0.022681      0.0305134     0.0238764    0.00549014       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.78952       8.32316       13.3831       7.32406       3.12959        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.2397        8.45526       14.9417       8.7439        3.62229        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.14298       7.74422       13.0341       7.58911       3.04003        10     
[art]:TriggerResults:TriggerResultInserter        3.0367e-05    5.19505e-05   0.000177865   3.9038e-05    4.22996e-05      10     
end_path:out1:RootOutput                           5.571e-06     7.356e-06    1.7122e-05     6.212e-06    3.28509e-06      10     
end_path:out1:RootOutput(write)                     2.06491       4.00124       5.63506       3.77075      0.988717        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5187.27 MB
  Peak resident set size usage (VmHWM): 4360.02 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.32 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.96 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.19 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10126040
-rw-r--r-- 1 dune01 dune01 2378935459 Jan 29 13:06 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2263671528 Jan 29 13:03 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2061089782 Jan 29 14:41 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1836229660 Jan 29 10:03 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1754939350 Jan 29 09:45 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   73571430 Jan 29 14:33 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     243098 Jan 29 14:41 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Jan 29 14:41 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Jan 29 14:41 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Jan 29 14:41 time.db
-rw-r--r-- 1 dune01 dune01        519 Jan 29 13:06 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 29 10:03 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 29 14:41 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Jan 29 09:36 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Jan 29 14:41 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Jan 29 13:17 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Jan 29 13:06 debugprod.log
justIN time: 2026-02-04 15:17:00 UTC       justIN version: 01.06.00