justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 273626.1@dunegpschedd01.fnal.gov

Jobsub ID273626.1@dunegpschedd01.fnal.gov
Workflow ID11689
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-10 17:45:48
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2026-01-10 22:26:29
From worker nodeHostnamewn-lb-12.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-10 17:46:37
Input filesmonte-carlo-011689-000001
JobscriptExit code0
Real time4h (16669s)
CPU time4h (16481s = 98%)
Max RSS bytes4356292608 (4154 MiB)
Outputting started2026-01-10 22:24:28
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11689/1/001/reco_stage1.root
Finished2026-01-10 22:26:29
Saved logsjustin-logs:273626.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

l adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 125971 traces, 0 summary
Retagger: tagging trace set: gauss with 98356 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 98356 traces tagged "gauss"
FrameSaver: q=3.42224e+06 n=1796314 tag=gauss
wclsFrameSaver: saving 125971 traces tagged "wiener"
FrameSaver: q=3.59601e+06 n=1600451 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 27912 XUs and 21769 XVs -> 206 XUVs
C:0 T:1 40448 XUs and 37877 XVs -> 3563 XUVs
C:0 T:2 136148 XUs and 103572 XVs -> 5629 XUVs
C:0 T:3 62922 XUs and 38128 XVs -> 318 XUVs
C:0 T:4 14271 XUs and 16287 XVs -> 220 XUVs
C:0 T:5 65157 XUs and 55094 XVs -> 5709 XUVs
C:0 T:6 90129 XUs and 51546 XVs -> 3028 XUVs
C:0 T:7 38034 XUs and 18590 XVs -> 129 XUVs
C:0 T:8 10348 XUs and 9035 XVs -> 76 XUVs
C:0 T:9 70622 XUs and 74126 XVs -> 4613 XUVs
C:0 T:10 40201 XUs and 22700 XVs -> 4295 XUVs
C:0 T:11 145 XUs and 99 XVs -> 16 XUVs
27802 XUVs total
11574 collection wire objects
26051 potential space points
Neighbour search...
1533529 tests to find 821052 neighbours
Iterating with no regularization...
Begin: 6.13437e+07
0 5.08124e+07
1 5.01838e+07
2 5.01025e+07
3 5.00788e+07
Now with regularization...
Begin: 4.1483e+07
0 4.14575e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411660134
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411660134
fLifetime: 30000 [us]
10-Jan-2026 23:24:25 CET  Closed output file "reco_stage1.root"
10-Jan-2026 23:24:25 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          362.983       457.36        631.746       429.055       82.754         10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00121471    0.00576783     0.010519     0.00586095    0.00257098       10     
reco:ophit:OpHitFinder                             0.381585      0.442331      0.503685      0.453916      0.0378232       10     
reco:opflash:OpFlashFinder                         0.264486      0.337627      0.414809      0.347364      0.0489548       10     
reco:opslicer:OpSlicer                             0.753043      0.908932       1.02594      0.933743      0.0890832       10     
reco:caldata:DataPrepModule                         6.33467       6.59345       7.18956       6.52417       0.22935        10     
reco:wclsdatasp:WireCellToolkit                     46.4322       61.3079       71.1969       63.9767       8.09007        10     
reco:gaushit:GausHitFinder                          1.2067        1.4046        1.79317       1.32181      0.190036        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000430214   0.000645456   0.00147421    0.000573087   0.000287325      10     
reco:reco3d:SpacePointSolver                        12.5096       21.9055       40.585        20.3952       8.22364        10     
reco:hitpdune:DisambigFromSpacePoints               1.29496       1.84968       2.80575       1.71516      0.479562        10     
reco:pandora:StandardPandora                        25.2861       41.161        70.9507       38.4631       14.6082        10     
reco:pandoraWriter:StandardPandora                 0.421796      0.519675       0.65903      0.508312      0.0742414       10     
reco:pandoraTrack:LArPandoraTrackCreation           5.05802       6.76649       10.005        6.26452       1.53066        10     
reco:pandoraShower:LArPandoraShowerCreation         4.93844       7.84803       14.7862       7.08858       3.17863        10     
reco:pandoracalo:Calorimetry                        2.96983       3.82657       5.82398       3.6999       0.833068        10     
reco:pandoracalonosce:Calorimetry                   2.77835       3.56375       5.24214       3.41428      0.767109        10     
reco:pandorapid:Chi2ParticleID                    0.00565682    0.00664981    0.00754944     0.0066559    0.000566756      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.047911      0.0579256     0.0713586     0.0578781    0.00695502       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00566264    0.00632737    0.00743572    0.00620495    0.000492707      10     
reco:pandoraShowercalo:ShowerCalorimetry            5.14034       8.02833       13.9494       7.37952       2.7948         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       4.93518       7.76123       13.2959       7.14013       2.74973        10     
reco:emtrkmichelid:EmTrackMichelId                  174.431       211.086       269.702       206.385       28.3234        10     
reco:crttag:SingleCRTMatchingProducer              0.0170953     0.0219451     0.0306301     0.0209809    0.00438997       10     
reco:crtreco:TwoCRTMatchingProducer                0.0102986     0.0150308     0.0189029     0.0151939    0.00247208       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.932793       1.09112       1.39011       1.03965       0.15439        10     
reco:pandora2Track:LArPandoraTrackCreation          14.8247       21.1588       34.3863       18.9271       6.83309        10     
reco:pandora2calo:Calorimetry                       7.41856       10.8297       18.0549       9.80206       3.46765        10     
reco:pandora2calonosce:Calorimetry                  7.30028       10.2894       16.5429       9.24775       3.16069        10     
reco:pandora2pid:Chi2ParticleID                    0.0130157     0.0147528     0.0191694     0.0139672    0.00209565       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0622509     0.0737747     0.0933523     0.0717321     0.0106445       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0129479     0.0146648     0.0188527     0.0138495    0.00202671       10     
reco:pandora2Shower:LArPandoraShowerCreation        5.1044        7.50094       11.8314       6.76557       2.25444        10     
reco:pandora2Showercalo:ShowerCalorimetry           6.13889       9.15058       14.1626       8.42472       2.51824        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      5.2753        7.88018       11.634        7.55584       1.9939         10     
[art]:TriggerResults:TriggerResultInserter        4.3405e-05    5.51005e-05   0.000130565   4.6215e-05    2.54025e-05      10     
end_path:out1:RootOutput                           7.671e-06     9.255e-06    1.6465e-05    8.5125e-06    2.42957e-06      10     
end_path:out1:RootOutput(write)                     2.99131       3.67536       4.73973       3.54775      0.502243        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5160.17 MB
  Peak resident set size usage (VmHWM): 4356.29 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.29 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.10 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10685148
-rw-r--r--. 1 dune009 dune 2483269052 Jan 10 22:05 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2353361292 Jan 10 22:03 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2186645556 Jan 10 23:24 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1958418039 Jan 10 19:10 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1880641668 Jan 10 18:54 g4_stage1.root
-rw-r--r--. 1 dune009 dune   78637480 Jan 10 23:19 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     243009 Jan 10 23:24 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Jan 10 23:24 mem.db
-rw-r--r--. 1 dune009 dune      62376 Jan 10 23:24 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Jan 10 23:24 time.db
-rw-r--r--. 1 dune009 dune        519 Jan 10 22:05 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 10 19:10 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 10 23:24 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Jan 10 18:46 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Jan 10 23:24 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Jan 10 22:13 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Jan 10 22:05 debugprod.log
justIN time: 2026-02-06 11:40:59 UTC       justIN version: 01.06.00