justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 251504.0@dunegpschedd01.fnal.gov

Jobsub ID251504.0@dunegpschedd01.fnal.gov
Workflow ID10213
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-15 08:06:33
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-11-15 11:28:43
From worker nodeHostnamewn-la-12.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-15 08:07:26
Input filesmonte-carlo-010213-000001
JobscriptExit code0
Real time3h (11986s)
CPU time3h (11771s = 98%)
Max RSS bytes4054925312 (3867 MiB)
Outputting started2025-11-15 11:27:14
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10213/1/001/reco_stage1.root
Finished2025-11-15 11:28:43
Saved logsjustin-logs:251504.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

g tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 119606 traces, 0 summary
Retagger: tagging trace set: gauss with 94053 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 94053 traces tagged "gauss"
FrameSaver: q=3.0447e+06 n=1684861 tag=gauss
wclsFrameSaver: saving 119606 traces tagged "wiener"
FrameSaver: q=3.20214e+06 n=1498783 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 44441 XUs and 37342 XVs -> 463 XUVs
C:0 T:1 106969 XUs and 92595 XVs -> 7048 XUVs
C:0 T:2 83038 XUs and 107245 XVs -> 5053 XUVs
C:0 T:3 36914 XUs and 45053 XVs -> 288 XUVs
C:0 T:4 9998 XUs and 9028 XVs -> 77 XUVs
C:0 T:5 34229 XUs and 22805 XVs -> 3153 XUVs
C:0 T:6 110523 XUs and 98862 XVs -> 3694 XUVs
C:0 T:7 42257 XUs and 37527 XVs -> 279 XUVs
C:0 T:8 6254 XUs and 5346 XVs -> 47 XUVs
C:0 T:9 35616 XUs and 39016 XVs -> 2835 XUVs
C:0 T:10 23347 XUs and 20787 XVs -> 2345 XUVs
C:0 T:11 200 XUs and 142 XVs -> 29 XUVs
25311 XUVs total
11391 collection wire objects
23213 potential space points
Neighbour search...
1484781 tests to find 719350 neighbours
Iterating with no regularization...
Begin: 5.14985e+07
0 4.3571e+07
1 4.29594e+07
2 4.2872e+07
3 4.28469e+07
Now with regularization...
Begin: 3.49655e+07
0 3.49464e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410525505
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410525505
fLifetime: 30000 [us]
15-Nov-2025 12:27:12 CET  Closed output file "reco_stage1.root"
15-Nov-2025 12:27:12 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          272.488       325.073       372.246       316.661       31.6317        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000992146   0.00448665    0.00785274    0.00446386    0.00223578       10     
reco:ophit:OpHitFinder                             0.267521      0.302967      0.330456      0.299147      0.0199065       10     
reco:opflash:OpFlashFinder                         0.148182      0.189955      0.227397       0.19523      0.0228811       10     
reco:opslicer:OpSlicer                             0.536494      0.602371      0.691221      0.594293      0.0516905       10     
reco:caldata:DataPrepModule                         4.76086       4.90959       5.2656        4.8827       0.133197        10     
reco:wclsdatasp:WireCellToolkit                     33.6427       43.2676       48.2533       45.559        5.07471        10     
reco:gaushit:GausHitFinder                         0.900094       1.00077       1.14376      0.974406      0.0937499       10     
reco:nhitsfilter:NumberOfHitsFilter               0.000342544   0.000462509   0.000735953   0.000386289   0.000137091      10     
reco:reco3d:SpacePointSolver                        9.72937       12.497        16.1897       12.0978       1.71285        10     
reco:hitpdune:DisambigFromSpacePoints              0.895732       1.21229       1.6102        1.14024       0.2082         10     
reco:pandora:StandardPandora                        17.6269       28.5736       42.4684       28.8595       7.14404        10     
reco:pandoraWriter:StandardPandora                 0.343289      0.383663      0.444798      0.379893      0.0325011       10     
reco:pandoraTrack:LArPandoraTrackCreation           4.04109       4.90087       6.7791        4.53357      0.785547        10     
reco:pandoraShower:LArPandoraShowerCreation          4.295        5.92997       7.60479       5.69274      0.932619        10     
reco:pandoracalo:Calorimetry                        1.97118       2.69935       3.73804       2.74569      0.474093        10     
reco:pandoracalonosce:Calorimetry                   1.91667       2.65779       3.94209       2.57623      0.554668        10     
reco:pandorapid:Chi2ParticleID                     0.0039673    0.00486125    0.00558805    0.00493124    0.000399284      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0350817     0.0414794     0.045423      0.0410594     0.0030493       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00400595    0.00459001    0.00508659    0.00465707    0.000284242      10     
reco:pandoraShowercalo:ShowerCalorimetry            4.31172       5.65567       7.4896        5.38244       1.07669        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       4.01214       5.52509       8.05654       5.13727       1.14143        10     
reco:emtrkmichelid:EmTrackMichelId                  129.613       150.357       173.106       146.827       13.2437        10     
reco:crttag:SingleCRTMatchingProducer             0.00947192     0.0156769     0.0283651     0.014966     0.00482916       10     
reco:crtreco:TwoCRTMatchingProducer               0.00837865     0.011683      0.0177675     0.0113602    0.00238088       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.632447      0.752101      0.903628      0.749267      0.0701747       10     
reco:pandora2Track:LArPandoraTrackCreation          12.5085       16.7675       20.9679       17.0155       2.53671        10     
reco:pandora2calo:Calorimetry                       5.56407       8.38176       11.2138       8.00288       1.65988        10     
reco:pandora2calonosce:Calorimetry                  5.7906        7.55552       9.60429       7.55278       1.12151        10     
reco:pandora2pid:Chi2ParticleID                   0.00925502     0.010453      0.0127865     0.0100682    0.000984851      10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0424987     0.0564238     0.0861194     0.0516956     0.0114124       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00941379     0.0105973     0.0129711     0.0103349    0.00109027       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.47629       6.20205       7.77015       6.23552       1.02529        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.70319       6.51448       8.17789       6.53261      0.884321        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      4.27102        5.391        6.62832       5.34677      0.733098        10     
[art]:TriggerResults:TriggerResultInserter        3.0005e-05    5.56492e-05   0.00012747    4.44415e-05   2.82227e-05      10     
end_path:out1:RootOutput                           6.801e-06    8.4191e-06    1.3651e-05    7.3665e-06    2.08549e-06      10     
end_path:out1:RootOutput(write)                     2.13194       2.47776       2.86612       2.38073      0.219995        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4854.5 MB
  Peak resident set size usage (VmHWM): 4054.93 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.22 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.05 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.13 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10000100
-rw-r--r--. 1 dune009 dune 2345867562 Nov 15 11:31 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2227899013 Nov 15 11:29 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2020912615 Nov 15 12:27 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1823790043 Nov 15 09:25 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1748875392 Nov 15 09:13 g4_stage1.root
-rw-r--r--. 1 dune009 dune   72143520 Nov 15 12:23 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242811 Nov 15 12:27 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Nov 15 12:27 mem.db
-rw-r--r--. 1 dune009 dune      62376 Nov 15 12:27 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Nov 15 12:27 time.db
-rw-r--r--. 1 dune009 dune        519 Nov 15 11:31 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov 15 09:25 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov 15 12:27 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Nov 15 09:07 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Nov 15 12:27 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Nov 15 11:36 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Nov 15 11:31 debugprod.log
justIN time: 2026-02-09 12:25:51 UTC       justIN version: 01.06.00