justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 243134.0@dunegpschedd01.fnal.gov

Jobsub ID243134.0@dunegpschedd01.fnal.gov
Workflow ID9602
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-04 19:30:27
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2025-11-05 00:39:37
From worker nodeHostnamewn-lb-15.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-04 19:31:46
Input filesmonte-carlo-009602-000001
JobscriptExit code0
Real time5h (18373s)
CPU time5h (18263s = 99%)
Max RSS bytes4500303872 (4291 MiB)
Outputting started2025-11-05 00:38:00
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09602/1/001/reco_stage1.root
Finished2025-11-05 00:39:37
Saved logsjustin-logs:243134.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

dc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 151027 traces, 0 summary
Retagger: tagging trace set: gauss with 114600 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 114600 traces tagged "gauss"
FrameSaver: q=4.08542e+06 n=2015845 tag=gauss
wclsFrameSaver: saving 151027 traces tagged "wiener"
FrameSaver: q=4.32828e+06 n=1775262 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 31923 XUs and 28451 XVs -> 255 XUVs
C:0 T:1 52433 XUs and 53940 XVs -> 2910 XUVs
C:0 T:2 98563 XUs and 160583 XVs -> 7225 XUVs
C:0 T:3 50036 XUs and 59853 XVs -> 625 XUVs
C:0 T:4 12542 XUs and 8590 XVs -> 59 XUVs
C:0 T:5 43200 XUs and 25651 XVs -> 3324 XUVs
C:0 T:6 111361 XUs and 148625 XVs -> 6530 XUVs
C:0 T:7 41114 XUs and 52609 XVs -> 440 XUVs
C:0 T:8 11061 XUs and 9597 XVs -> 274 XUVs
C:0 T:9 117293 XUs and 97795 XVs -> 21839 XUVs
C:0 T:10 72185 XUs and 45123 XVs -> 8303 XUVs
C:0 T:11 36 XUs and 21 XVs -> 3 XUVs
51787 XUVs total
13134 collection wire objects
48412 potential space points
Neighbour search...
7457676 tests to find 3359010 neighbours
Iterating with no regularization...
Begin: 8.1153e+07
0 6.77591e+07
1 6.66186e+07
2 6.64556e+07
3 6.64073e+07
Now with regularization...
Begin: 5.47899e+07
0 5.47309e+07
1 5.4687e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410313780
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410313780
fLifetime: 30000 [us]
05-Nov-2025 01:37:58 CET  Closed output file "reco_stage1.root"
05-Nov-2025 01:37:58 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          450.399       551.827       627.587       567.073       50.9519        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00141845    0.00722644     0.0131808    0.00764794    0.00357917       10     
reco:ophit:OpHitFinder                             0.402039      0.461104      0.487723       0.47507      0.0284056       10     
reco:opflash:OpFlashFinder                         0.266991      0.357341      0.424584      0.360491      0.0496041       10     
reco:opslicer:OpSlicer                             0.755231       0.91156       1.01002      0.914602      0.0810534       10     
reco:caldata:DataPrepModule                         6.40323       6.58358       7.13143       6.48961      0.227692        10     
reco:wclsdatasp:WireCellToolkit                     46.4397       66.8892       74.9084       72.5654       10.5658        10     
reco:gaushit:GausHitFinder                          1.33823       1.52959       1.74031       1.56009      0.124161        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000631635   0.000912388    0.0012082    0.000883145   0.000151122      10     
reco:reco3d:SpacePointSolver                        16.5641       24.5626       48.8668       22.7021       8.98634        10     
reco:hitpdune:DisambigFromSpacePoints               1.50905       2.06236       2.48204       2.10688      0.250181        10     
reco:pandora:StandardPandora                        35.0845       46.7085       61.7521       45.9638       7.75049        10     
reco:pandoraWriter:StandardPandora                 0.473607      0.603365      0.691518      0.605343      0.0592158       10     
reco:pandoraTrack:LArPandoraTrackCreation           7.66612       10.368        14.0277       10.6153       1.89586        10     
reco:pandoraShower:LArPandoraShowerCreation         8.06046       12.5672       18.5921       12.0571       3.14718        10     
reco:pandoracalo:Calorimetry                        4.05479       5.71792       9.92923       5.01544       1.71691        10     
reco:pandoracalonosce:Calorimetry                   3.87553       5.41736       10.2445       4.55907       1.86351        10     
reco:pandorapid:Chi2ParticleID                    0.00630514    0.00749605    0.00982628    0.00742006    0.00101203       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0547215     0.0633762     0.0796957     0.0628233    0.00625754       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00624761    0.00714117    0.00846858    0.00690405    0.000749803      10     
reco:pandoraShowercalo:ShowerCalorimetry            7.54582       12.8963       19.4426       13.2517       3.2038         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       7.46833       11.5361       14.8079       11.5238       2.04232        10     
reco:emtrkmichelid:EmTrackMichelId                  198.095       241.108       263.566       242.883       18.1745        10     
reco:crttag:SingleCRTMatchingProducer              0.0168172     0.0261424     0.0500715     0.0223147    0.00986618       10     
reco:crtreco:TwoCRTMatchingProducer                0.0110546     0.020192      0.0307243     0.0193585    0.00592981       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.971116       1.21642       1.42457        1.23         0.11005        10     
reco:pandora2Track:LArPandoraTrackCreation          19.3236       30.9823       35.551        33.0012       4.8197         10     
reco:pandora2calo:Calorimetry                       9.45981       15.0439       19.2253       15.0313       3.24165        10     
reco:pandora2calonosce:Calorimetry                  9.53723       14.9135       19.6908       15.0118       2.81639        10     
reco:pandora2pid:Chi2ParticleID                    0.0131695     0.0159172     0.0173149     0.0164168    0.00137467       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0654383     0.0796203     0.0913794     0.0803018    0.00671698       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0128251     0.0158846     0.0172806     0.0162677    0.00144265       10     
reco:pandora2Shower:LArPandoraShowerCreation        8.0733        11.7622       15.081        11.9292       2.13148        10     
reco:pandora2Showercalo:ShowerCalorimetry           9.08741       12.6431        16.91        12.5944       2.77831        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      7.84271       10.6201       14.8461       10.5199       2.16311        10     
[art]:TriggerResults:TriggerResultInserter        3.9137e-05    6.95588e-05   0.00016753    5.80215e-05   3.50012e-05      10     
end_path:out1:RootOutput                           8.202e-06    1.03226e-05   2.0201e-05     8.909e-06    3.43877e-06      10     
end_path:out1:RootOutput(write)                     3.20754       3.80161       4.15781       3.83924      0.269362        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5304.76 MB
  Peak resident set size usage (VmHWM): 4500.3 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.29 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.08 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11203600
-rw-r--r--. 1 dune009 dune 2588203943 Nov  5 00:03 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2450459449 Nov  5 00:01 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2309000988 Nov  5 01:37 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2064959526 Nov  4 20:57 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1976307368 Nov  4 20:39 g4_stage1.root
-rw-r--r--. 1 dune009 dune   82945300 Nov  5 01:31 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242920 Nov  5 01:38 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Nov  5 01:37 mem.db
-rw-r--r--. 1 dune009 dune      62376 Nov  5 01:37 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Nov  5 01:37 time.db
-rw-r--r--. 1 dune009 dune        519 Nov  5 00:03 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov  4 20:57 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov  5 01:37 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Nov  4 20:31 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Nov  5 01:37 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Nov  5 00:11 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Nov  5 00:03 debugprod.log
justIN time: 2026-02-11 07:40:27 UTC       justIN version: 01.06.00