justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 279647.1@dunegpschedd01.fnal.gov

Jobsub ID279647.1@dunegpschedd01.fnal.gov
Workflow ID11992
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-16 21:23:37
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2026-01-17 01:46:43
From worker nodeHostnamewn-la-22.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-16 21:24:31
Input filesmonte-carlo-011992-000001
JobscriptExit code0
Real time4h (15623s)
CPU time4h (15478s = 99%)
Max RSS bytes4447162368 (4241 MiB)
Outputting started2026-01-17 01:44:55
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11992/1/001/reco_stage1.root
Finished2026-01-17 01:46:43
Saved logsjustin-logs:279647.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 131473 traces, 0 summary
Retagger: tagging trace set: gauss with 102294 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 102294 traces tagged "gauss"
FrameSaver: q=3.61412e+06 n=1855887 tag=gauss
wclsFrameSaver: saving 131473 traces tagged "wiener"
FrameSaver: q=3.82073e+06 n=1658293 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 21354 XUs and 21422 XVs -> 366 XUVs
C:0 T:1 85911 XUs and 50591 XVs -> 7603 XUVs
C:0 T:2 77273 XUs and 74639 XVs -> 3142 XUVs
C:0 T:3 41171 XUs and 30443 XVs -> 290 XUVs
C:0 T:4 15742 XUs and 14801 XVs -> 149 XUVs
C:0 T:5 72303 XUs and 56802 XVs -> 9016 XUVs
C:0 T:6 222993 XUs and 207459 XVs -> 10839 XUVs
C:0 T:7 85669 XUs and 75839 XVs -> 1133 XUVs
C:0 T:8 7143 XUs and 5065 XVs -> 46 XUVs
C:0 T:9 35171 XUs and 39954 XVs -> 2246 XUVs
C:0 T:10 29446 XUs and 36083 XVs -> 4293 XUVs
C:0 T:11 447 XUs and 352 XVs -> 70 XUVs
39193 XUVs total
11711 collection wire objects
35931 potential space points
Neighbour search...
3670609 tests to find 1704572 neighbours
Iterating with no regularization...
Begin: 7.67033e+07
0 5.95949e+07
1 5.83643e+07
2 5.8198e+07
3 5.81556e+07
Now with regularization...
Begin: 4.74761e+07
0 4.7425e+07
1 4.74004e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411783832
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411783832
fLifetime: 30000 [us]
17-Jan-2026 02:44:53 CET  Closed output file "reco_stage1.root"
17-Jan-2026 02:44:53 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          389.655       543.409       701.274       536.625       97.7819        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000908899   0.00563546     0.010275     0.00551582    0.00297023       10     
reco:ophit:OpHitFinder                             0.375642      0.430178      0.483833      0.433688      0.0367397       10     
reco:opflash:OpFlashFinder                         0.224382      0.299282      0.364612      0.297723      0.0416549       10     
reco:opslicer:OpSlicer                             0.682297      0.894321       1.0678       0.911927       0.1244         10     
reco:caldata:DataPrepModule                         5.36862       6.33072       6.64149       6.41288      0.338415        10     
reco:wclsdatasp:WireCellToolkit                     35.8853       57.7873       64.9505       62.0072       9.02179        10     
reco:gaushit:GausHitFinder                          1.25717       1.56878       1.99921       1.49285      0.269661        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000427458   0.000685044    0.0011767    0.000584132   0.000268623      10     
reco:reco3d:SpacePointSolver                        10.7782       24.763        38.2614       22.8577       7.47902        10     
reco:hitpdune:DisambigFromSpacePoints               1.51049       2.34538       3.26758       2.30337      0.557758        10     
reco:pandora:StandardPandora                        29.1895       61.0867       106.206       55.1614       25.3443        10     
reco:pandoraWriter:StandardPandora                 0.428674      0.564254      0.695576      0.534529      0.0873229       10     
reco:pandoraTrack:LArPandoraTrackCreation           4.17318       8.68829       15.6725       7.8652        2.98378        10     
reco:pandoraShower:LArPandoraShowerCreation         4.87225       11.2009       22.5394       10.4727       4.92855        10     
reco:pandoracalo:Calorimetry                        2.38606       4.72109       8.84421       4.30078       1.65089        10     
reco:pandoracalonosce:Calorimetry                   2.24424       4.5593        8.16967       4.36254       1.54254        10     
reco:pandorapid:Chi2ParticleID                    0.00548171    0.00696786    0.00797793     0.0071161    0.000750864      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0483041     0.0614458     0.0758458     0.0607586    0.00777729       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00537543    0.00668232     0.0078507     0.0067414    0.000765782      10     
reco:pandoraShowercalo:ShowerCalorimetry            5.28376       11.2639       20.1421        10.47        4.50739        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       4.8268        10.4713       18.6902       10.042        3.98932        10     
reco:emtrkmichelid:EmTrackMichelId                  202.22        247.518       309.565       242.463       33.9915        10     
reco:crttag:SingleCRTMatchingProducer              0.0127649     0.0215125     0.0270718     0.0218441    0.00476829       10     
reco:crtreco:TwoCRTMatchingProducer                0.011133      0.0163458     0.0206943     0.0168891    0.00313824       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.845459        1.144        1.45403       1.14626      0.156347        10     
reco:pandora2Track:LArPandoraTrackCreation          14.1074       27.8788       37.932        28.1411       7.74465        10     
reco:pandora2calo:Calorimetry                       7.4749        13.3775       17.3835       13.2484       3.27486        10     
reco:pandora2calonosce:Calorimetry                  6.96094       12.7239       16.6288       12.9307       2.93271        10     
reco:pandora2pid:Chi2ParticleID                    0.012918      0.0167257     0.0200444      0.01742     0.00231052       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0638105     0.0825545     0.0996224     0.080642      0.0112841       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0128763     0.0167804     0.019925      0.017403     0.00233261       10     
reco:pandora2Shower:LArPandoraShowerCreation        5.08493       9.48747       12.6794       9.46741       2.47677        10     
reco:pandora2Showercalo:ShowerCalorimetry           6.11306       10.6724       17.0992       10.1835       3.01266        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      5.82592       9.13884       15.0342       8.74654       2.67842        10     
[art]:TriggerResults:TriggerResultInserter        3.7015e-05    5.29969e-05   0.000114642   4.6524e-05    2.14889e-05      10     
end_path:out1:RootOutput                            7.1e-06     8.1791e-06    1.2148e-05    7.7265e-06    1.38383e-06      10     
end_path:out1:RootOutput(write)                     3.15487       4.00595       4.93418       3.92064      0.610953        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5247.77 MB
  Peak resident set size usage (VmHWM): 4447.16 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.28 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.99 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.16 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11088636
-rw-r--r--. 1 dune009 dune 2585296717 Jan 17 01:12 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2452565015 Jan 17 01:10 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2329456648 Jan 17 02:44 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1992316272 Jan 16 22:48 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1905769590 Jan 16 22:32 g4_stage1.root
-rw-r--r--. 1 dune009 dune   88746700 Jan 17 02:38 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     243237 Jan 17 02:44 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Jan 17 02:44 mem.db
-rw-r--r--. 1 dune009 dune      62376 Jan 17 02:44 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Jan 17 02:44 time.db
-rw-r--r--. 1 dune009 dune        519 Jan 17 01:12 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 16 22:48 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 17 02:44 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Jan 16 22:24 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Jan 17 02:44 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Jan 17 01:20 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Jan 17 01:12 debugprod.log
justIN time: 2026-02-05 23:27:21 UTC       justIN version: 01.06.00