justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 243691.0@dunegpschedd01.fnal.gov

Jobsub ID243691.0@dunegpschedd01.fnal.gov
Workflow ID9643
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-05 20:01:47
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-11-06 01:06:59
From worker nodeHostnamewn-da-22.gina.surf.nl
cpuinfoAMD EPYC-Rome Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-05 20:03:11
Input filesmonte-carlo-009643-000001
JobscriptExit code0
Real time5h (18094s)
CPU time4h (17864s = 98%)
Max RSS bytes4637208576 (4422 MiB)
Outputting started2025-11-06 01:04:47
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09643/1/001/reco_stage1.root
Finished2025-11-06 01:06:59
Saved logsjustin-logs:243691.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

Running tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 139245 traces, 0 summary
Retagger: tagging trace set: gauss with 108221 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 108221 traces tagged "gauss"
FrameSaver: q=4.0163e+06 n=1962597 tag=gauss
wclsFrameSaver: saving 139245 traces tagged "wiener"
FrameSaver: q=4.25625e+06 n=1762438 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 45430 XUs and 47765 XVs -> 1018 XUVs
C:0 T:1 95720 XUs and 97112 XVs -> 11101 XUVs
C:0 T:2 80545 XUs and 121969 XVs -> 5771 XUVs
C:0 T:3 37476 XUs and 40064 XVs -> 231 XUVs
C:0 T:4 11632 XUs and 11090 XVs -> 82 XUVs
C:0 T:5 39331 XUs and 35880 XVs -> 3462 XUVs
C:0 T:6 140679 XUs and 219751 XVs -> 20700 XUVs
C:0 T:7 50159 XUs and 56944 XVs -> 947 XUVs
C:0 T:9 31885 XUs and 54165 XVs -> 7931 XUVs
C:0 T:10 54927 XUs and 37653 XVs -> 4738 XUVs
C:0 T:11 814 XUs and 580 XVs -> 96 XUVs
56077 XUVs total
11899 collection wire objects
50576 potential space points
Neighbour search...
9129982 tests to find 3847940 neighbours
Iterating with no regularization...
Begin: 1.19915e+08
0 9.26552e+07
1 9.05007e+07
2 9.02161e+07
3 9.0127e+07
Now with regularization...
Begin: 7.40154e+07
0 7.38672e+07
1 7.37876e+07
2 7.37307e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410334171
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410334171
fLifetime: 30000 [us]
06-Nov-2025 02:04:44 CET  Closed output file "reco_stage1.root"
06-Nov-2025 02:04:44 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          458.662       642.527       830.288       652.856       103.172        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00147562    0.00724048     0.0114323    0.00721463     0.0033921       10     
reco:ophit:OpHitFinder                             0.326866      0.365586      0.411675      0.369506      0.0280901       10     
reco:opflash:OpFlashFinder                         0.218137      0.275287      0.350258      0.269846      0.0420976       10     
reco:opslicer:OpSlicer                             0.731805      0.858906       0.9592       0.894536      0.0881398       10     
reco:caldata:DataPrepModule                         5.77395       5.91253       6.30643       5.82955      0.172456        10     
reco:wclsdatasp:WireCellToolkit                     44.9588       67.6888       77.1821       73.4075       11.2226        10     
reco:gaushit:GausHitFinder                          1.2639        1.74491       2.29797       1.74763      0.272528        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000753977   0.00175895    0.00304722    0.00167993    0.000633892      10     
reco:reco3d:SpacePointSolver                        15.7112       23.7737       40.2537       22.4988       6.69152        10     
reco:hitpdune:DisambigFromSpacePoints               1.49159       2.32913       3.35881       2.32756      0.542776        10     
reco:pandora:StandardPandora                        29.0939       55.0715       96.5255       52.6974       17.4474        10     
reco:pandoraWriter:StandardPandora                 0.557505       0.68463      0.817818      0.681548      0.0849426       10     
reco:pandoraTrack:LArPandoraTrackCreation           9.25209       13.2168       19.2934       12.6475       2.79477        10     
reco:pandoraShower:LArPandoraShowerCreation         8.9423        16.5861       24.5662       16.2748       4.82944        10     
reco:pandoracalo:Calorimetry                        4.3508        6.81674       9.80687       7.05681       1.54038        10     
reco:pandoracalonosce:Calorimetry                   4.24555       6.52413       9.41059       6.6163        1.51102        10     
reco:pandorapid:Chi2ParticleID                    0.00710396    0.00890608     0.0113039    0.00896471     0.001114        10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0557564      0.06761      0.079973      0.0693443     0.0079308       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00648626    0.00805705     0.0100853    0.00787397    0.00105118       10     
reco:pandoraShowercalo:ShowerCalorimetry            8.67777       15.9891       22.8186       16.804        4.1613         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       8.15613       15.3113       23.5568       15.3616       4.18738        10     
reco:emtrkmichelid:EmTrackMichelId                  224.22        277.85        333.988       278.148       34.0222        10     
reco:crttag:SingleCRTMatchingProducer              0.0223354     0.0348378     0.0549113     0.0344898     0.0085499       10     
reco:crtreco:TwoCRTMatchingProducer                0.0158813     0.0270867     0.0340139     0.0285479    0.00467599       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.905047       1.08359       1.31589       1.11079      0.126774        10     
reco:pandora2Track:LArPandoraTrackCreation          27.0808       42.8002       60.2209       42.7969       10.2511        10     
reco:pandora2calo:Calorimetry                       13.1224       21.013        28.2079       21.736        4.8926         10     
reco:pandora2calonosce:Calorimetry                  12.8246       19.5509       26.1159       20.0621       4.1349         10     
reco:pandora2pid:Chi2ParticleID                    0.0165279     0.0207502     0.0247994     0.0203662    0.00290398       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0678567     0.0882696      0.10453      0.0899314     0.011312        10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0146863     0.0201612     0.0242978     0.0204311    0.00286386       10     
reco:pandora2Shower:LArPandoraShowerCreation        8.98498       14.0489       18.9934       14.0002       2.82463        10     
reco:pandora2Showercalo:ShowerCalorimetry           10.0836       15.8993       20.4036       16.1492       3.03266        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      7.84477       12.5809       16.319        13.4069       2.56736        10     
[art]:TriggerResults:TriggerResultInserter        4.3143e-05    5.73422e-05   0.000118325   5.03205e-05   2.10556e-05      10     
end_path:out1:RootOutput                           7.625e-06    9.5495e-06    1.4428e-05     9.207e-06    1.83192e-06      10     
end_path:out1:RootOutput(write)                     3.26202       3.97517       4.86006       3.90479      0.507417        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5439.16 MB
  Peak resident set size usage (VmHWM): 4637.21 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.11 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.27 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.56 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.13 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11767960
-rw-r--r--. 1 dune009 dune 2705913395 Nov  6 00:14 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2566931046 Nov  6 00:11 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2440960042 Nov  6 02:04 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2171993337 Nov  5 21:29 g4_stage2.root
-rw-r--r--. 1 dune009 dune 2077579276 Nov  5 21:11 g4_stage1.root
-rw-r--r--. 1 dune009 dune   86331980 Nov  6 01:56 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     243316 Nov  6 02:04 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Nov  6 02:04 mem.db
-rw-r--r--. 1 dune009 dune      62376 Nov  6 02:04 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Nov  6 02:04 time.db
-rw-r--r--. 1 dune009 dune        519 Nov  6 00:14 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov  5 21:29 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov  6 02:04 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Nov  5 21:03 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Nov  6 02:04 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Nov  6 00:21 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Nov  6 00:14 debugprod.log
justIN time: 2026-02-11 06:07:31 UTC       justIN version: 01.06.00