justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 271484.0@dunegpschedd02.fnal.gov

Jobsub ID271484.0@dunegpschedd02.fnal.gov
Workflow ID11832
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-13 15:41:29
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2026-01-13 20:12:32
From worker nodeHostnamewn-la-04.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-13 15:42:31
Input filesmonte-carlo-011832-000001
JobscriptExit code0
Real time4h (16082s)
CPU time4h (15878s = 98%)
Max RSS bytes4186124288 (3992 MiB)
Outputting started2026-01-13 20:10:34
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11832/1/001/reco_stage1.root
Finished2026-01-13 20:12:32
Saved logsjustin-logs:271484.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 175026 traces, 0 summary
Retagger: tagging trace set: gauss with 136018 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 136018 traces tagged "gauss"
FrameSaver: q=4.42838e+06 n=2417182 tag=gauss
wclsFrameSaver: saving 175026 traces tagged "wiener"
FrameSaver: q=4.68372e+06 n=2153851 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 39885 XUs and 37178 XVs -> 411 XUVs
C:0 T:1 72512 XUs and 67526 XVs -> 4971 XUVs
C:0 T:2 201851 XUs and 202630 XVs -> 10296 XUVs
C:0 T:3 83979 XUs and 76788 XVs -> 968 XUVs
C:0 T:4 14363 XUs and 16502 XVs -> 241 XUVs
C:0 T:5 65800 XUs and 79575 XVs -> 10886 XUVs
C:0 T:6 184428 XUs and 190754 XVs -> 10881 XUVs
C:0 T:7 66360 XUs and 59740 XVs -> 825 XUVs
C:0 T:8 14236 XUs and 12177 XVs -> 242 XUVs
C:0 T:9 98766 XUs and 106818 XVs -> 6995 XUVs
C:0 T:10 22888 XUs and 9987 XVs -> 2017 XUVs
C:0 T:11 241 XUs and 161 XVs -> 23 XUVs
48756 XUVs total
15718 collection wire objects
44148 potential space points
Neighbour search...
3678504 tests to find 1847316 neighbours
Iterating with no regularization...
Begin: 8.14958e+07
0 6.85781e+07
1 6.75387e+07
2 6.74014e+07
3 6.73609e+07
Now with regularization...
Begin: 5.57321e+07
0 5.57028e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411718715
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411718715
fLifetime: 30000 [us]
13-Jan-2026 21:10:31 CET  Closed output file "reco_stage1.root"
13-Jan-2026 21:10:31 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          289.036       430.887       652.552       413.366       96.3618        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00115035    0.00572218    0.00944204     0.0061863    0.00270181       10     
reco:ophit:OpHitFinder                              0.34681       0.42828      0.516906      0.428524      0.0533977       10     
reco:opflash:OpFlashFinder                         0.227567      0.321289      0.435807      0.322024      0.0595659       10     
reco:opslicer:OpSlicer                             0.689977      0.858547       1.10883      0.844523      0.132437        10     
reco:caldata:DataPrepModule                         4.79406       6.31331       7.18473       6.39496       0.56573        10     
reco:wclsdatasp:WireCellToolkit                     42.0809       53.8428       62.9871       55.2479       6.98952        10     
reco:gaushit:GausHitFinder                         0.991226       1.36354       1.77692       1.35402      0.216031        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000374357   0.000559089   0.00136878    0.000459806   0.00027782       10     
reco:reco3d:SpacePointSolver                        11.7386       20.631        38.7177       19.2166       7.81957        10     
reco:hitpdune:DisambigFromSpacePoints               1.04529       1.84733       3.13691       1.73259       0.60288        10     
reco:pandora:StandardPandora                        19.5564       42.3675       78.1007       41.4455       15.6829        10     
reco:pandoraWriter:StandardPandora                 0.362845       0.48685      0.644259      0.471611      0.083293        10     
reco:pandoraTrack:LArPandoraTrackCreation           4.11369       6.28879       9.68284       6.14751       1.72818        10     
reco:pandoraShower:LArPandoraShowerCreation         2.96121       7.2462        16.2546       6.68082       3.5628         10     
reco:pandoracalo:Calorimetry                        1.89231       3.54289        6.383        3.48488       1.20787        10     
reco:pandoracalonosce:Calorimetry                   1.87963       3.36431       6.44922       3.21116       1.22976        10     
reco:pandorapid:Chi2ParticleID                    0.00566156    0.00661995    0.00908776    0.00640895    0.000940865      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0440099     0.0554478     0.070159      0.0542555    0.00752215       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00548209    0.00625239    0.00703882    0.00625544    0.000469842      10     
reco:pandoraShowercalo:ShowerCalorimetry            2.46262       7.62676       19.0616       7.34043       4.37568        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.40835       6.86486       14.7042       6.79972       3.29886        10     
reco:emtrkmichelid:EmTrackMichelId                  152.748       205.422       274.79        201.719       34.8187        10     
reco:crttag:SingleCRTMatchingProducer              0.0117767     0.0198663     0.0351646     0.019626     0.00622905       10     
reco:crtreco:TwoCRTMatchingProducer               0.00838824     0.012825      0.0176643     0.0135769    0.00270109       10     
reco:anodepiercerst0:T0RecoAnodePiercers            0.69031       1.06318       1.30717       1.10939      0.176504        10     
reco:pandora2Track:LArPandoraTrackCreation          8.57212       18.326        34.9384       17.1803       6.98057        10     
reco:pandora2calo:Calorimetry                       4.21531       9.19482       16.6689       8.81414       3.43596        10     
reco:pandora2calonosce:Calorimetry                  3.88665       8.82354       17.613        8.25181       3.82006        10     
reco:pandora2pid:Chi2ParticleID                    0.0113374     0.0142294     0.0181476     0.0141476    0.00218715       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0510249     0.0704119      0.10375      0.0677549     0.0142568       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0112684     0.0142536     0.0184877     0.0143316    0.00230426       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.34804       6.40355       12.5918       6.37556       2.62538        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.49209       7.78222       14.7278       7.44899       3.18915        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.17879       6.59347       11.5242       6.60086       2.46376        10     
[art]:TriggerResults:TriggerResultInserter        3.4481e-05    5.43901e-05   0.000147721   4.57725e-05   3.13528e-05      10     
end_path:out1:RootOutput                           6.529e-06    5.10742e-05   0.000441349    7.887e-06    0.000130093      10     
end_path:out1:RootOutput(write)                     2.49544       3.47633       4.75616       3.50198      0.708201        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5114.65 MB
  Peak resident set size usage (VmHWM): 4186.12 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.28 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.96 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.16 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10578056
-rw-r--r--. 1 dune009 dune 2472735039 Jan 13 19:56 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2342650160 Jan 13 19:54 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2170057354 Jan 13 21:10 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1925443398 Jan 13 17:06 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1843122283 Jan 13 16:50 g4_stage1.root
-rw-r--r--. 1 dune009 dune   77297350 Jan 13 21:02 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242875 Jan 13 21:10 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Jan 13 21:10 mem.db
-rw-r--r--. 1 dune009 dune      62376 Jan 13 21:10 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Jan 13 21:10 time.db
-rw-r--r--. 1 dune009 dune        519 Jan 13 19:56 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 13 17:06 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 13 21:10 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Jan 13 16:42 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Jan 13 21:10 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Jan 13 20:03 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Jan 13 19:56 debugprod.log
justIN time: 2026-02-06 05:56:44 UTC       justIN version: 01.06.00