justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 14108.0@dunegpschedd02.fnal.gov

Jobsub ID14108.0@dunegpschedd02.fnal.gov
Workflow ID266
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-08-03 07:30:14
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2025-08-03 11:37:02
From worker nodeHostnamecomp01-31
cpuinfoIntel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-03 07:31:31
Input filesmonte-carlo-000266-000001
JobscriptExit code0
Real time4h (14650s)
CPU time3h (13517s = 92%)
Max RSS bytes4069560320 (3881 MiB)
Outputting started2025-08-03 11:35:42
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/00266/1/001/reco_stage1.root
Finished2025-08-03 11:37:02
Saved logsjustin-logs:14108.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

epService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 118256 traces, 0 summary
Retagger: tagging trace set: gauss with 92393 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 92393 traces tagged "gauss"
FrameSaver: q=3.18095e+06 n=1652812 tag=gauss
wclsFrameSaver: saving 118256 traces tagged "wiener"
FrameSaver: q=3.36571e+06 n=1480942 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 52197 XUs and 44456 XVs -> 922 XUVs
C:0 T:1 91334 XUs and 86050 XVs -> 5806 XUVs
C:0 T:2 54525 XUs and 78499 XVs -> 5020 XUVs
C:0 T:3 22630 XUs and 30914 XVs -> 239 XUVs
C:0 T:4 102088 XUs and 10864 XVs -> 2331 XUVs
C:0 T:5 28422 XUs and 16184 XVs -> 2440 XUVs
C:0 T:6 123738 XUs and 173570 XVs -> 8584 XUVs
C:0 T:7 47265 XUs and 61071 XVs -> 653 XUVs
C:0 T:9 17188 XUs and 27380 XVs -> 1617 XUVs
C:0 T:10 38865 XUs and 27544 XVs -> 3960 XUVs
C:0 T:11 169 XUs and 150 XVs -> 18 XUVs
31590 XUVs total
10050 collection wire objects
28539 potential space points
Neighbour search...
2038187 tests to find 1111586 neighbours
Iterating with no regularization...
Begin: 8.14964e+07
0 6.50331e+07
1 6.39945e+07
2 6.38588e+07
3 6.38141e+07
Now with regularization...
Begin: 5.47386e+07
0 5.47048e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 408432859
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 408432859
fLifetime: 30000 [us]
03-Aug-2025 12:35:39 BST  Closed output file "reco_stage1.root"
03-Aug-2025 12:35:39 BST  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          378.911       472.13        602.811       459.767       75.5808        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00104059    0.00410688    0.00672933    0.00384359    0.00177736       10     
reco:ophit:OpHitFinder                             0.253915      0.294361      0.332272      0.297756      0.0257128       10     
reco:opflash:OpFlashFinder                         0.119197      0.182256      0.227078      0.184486      0.0333984       10     
reco:opslicer:OpSlicer                             0.580202      0.740201      0.890719       0.74132      0.0921076       10     
reco:caldata:DataPrepModule                         6.31147       6.53964       6.93809       6.4992       0.194079        10     
reco:wclsdatasp:WireCellToolkit                     61.9127       72.3446       82.6503       72.5541       4.94608        10     
reco:gaushit:GausHitFinder                          1.41681       1.70364        1.962        1.76422      0.186061        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000541526   0.000733877   0.00120295    0.000693399   0.000177021      10     
reco:reco3d:SpacePointSolver                        9.90673       18.7857       31.3414       17.7425       6.42165        10     
reco:hitpdune:DisambigFromSpacePoints               1.99104       2.73111       3.64886       2.5563       0.517382        10     
reco:pandora:StandardPandora                        25.3171       46.3437       64.4089       45.3948       10.742         10     
reco:pandoraWriter:StandardPandora                 0.452464      0.537853      0.617571      0.532576      0.0565957       10     
reco:pandoraTrack:LArPandoraTrackCreation           4.56242       8.59376       11.6816       8.49636       2.01472        10     
reco:pandoraShower:LArPandoraShowerCreation         4.62593       9.38519       15.5877       8.94063       3.41922        10     
reco:pandoracalo:Calorimetry                        2.52049       3.97033       5.6179        4.01187      0.880936        10     
reco:pandoracalonosce:Calorimetry                   2.29002       3.63378       5.3299        3.56734      0.864201        10     
reco:pandorapid:Chi2ParticleID                    0.00574708    0.00670082    0.00735684    0.00658658    0.000508813      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0676026     0.0789761     0.0922683     0.0798811    0.00763983       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00561188    0.00642521    0.00717191    0.00627967    0.000459088      10     
reco:pandoraShowercalo:ShowerCalorimetry            4.65418       8.44348       13.9865       8.04759       3.1501         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       4.43285       8.1597        13.6869       7.82779       3.09846        10     
reco:emtrkmichelid:EmTrackMichelId                  162.54        200.533       246.212       192.619       26.3083        10     
reco:crttag:SingleCRTMatchingProducer              0.0195489     0.0249665     0.0336393     0.023028     0.00462133       10     
reco:crtreco:TwoCRTMatchingProducer                0.0112179     0.0190955     0.0288733     0.019331     0.00515398       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.69942       2.01915       2.32952       2.01633      0.200388        10     
reco:pandora2Track:LArPandoraTrackCreation          15.1724       26.467        42.6522       25.7623       8.84995        10     
reco:pandora2calo:Calorimetry                       7.28231       11.208        18.438        9.98981       3.94231        10     
reco:pandora2calonosce:Calorimetry                  6.70288       10.3859       17.6732       9.20632       3.84622        10     
reco:pandora2pid:Chi2ParticleID                    0.0132744     0.0156858     0.0204045     0.0148976    0.00234672       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0838623     0.0979551     0.119721      0.0961988     0.0121534       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.013022      0.0154501     0.0199387     0.0148368    0.00219327       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.99043       8.08077       15.2977       7.28073       3.13635        10     
reco:pandora2Showercalo:ShowerCalorimetry           6.05091       9.28569       16.1237       8.12603       3.31744        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      5.15213       8.34161       14.8334       7.17407       3.0197         10     
[art]:TriggerResults:TriggerResultInserter        5.4385e-05    7.70718e-05   0.000137654   7.26875e-05   2.13462e-05      10     
end_path:out1:RootOutput                          1.0802e-05    1.53502e-05   1.9837e-05    1.47015e-05   2.69827e-06      10     
end_path:out1:RootOutput(write)                     2.42501       2.99119       3.58268       2.86636      0.383194        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4888.65 MB
  Peak resident set size usage (VmHWM): 4069.56 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.11 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.26 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.06 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.18 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9742216
-rw-r--r-- 1 pltdune002 pltdune 2298761427 Aug  3 11:13 detsim_stage2.root
-rw-r--r-- 1 pltdune002 pltdune 2182562249 Aug  3 11:11 detsim_stage1.root
-rw-r--r-- 1 pltdune002 pltdune 1974117709 Aug  3 12:35 reco_stage1.root
-rw-r--r-- 1 pltdune002 pltdune 1761801164 Aug  3 09:07 g4_stage2.root
-rw-r--r-- 1 pltdune002 pltdune 1686405492 Aug  3 08:38 g4_stage1.root
-rw-r--r-- 1 pltdune002 pltdune   71758850 Aug  3 12:31 Pandora_Events.pndr
-rw-r--r-- 1 pltdune002 pltdune     242553 Aug  3 12:35 jobscript.log
-rw-r--r-- 1 pltdune002 pltdune     217088 Aug  3 12:35 mem.db
-rw-r--r-- 1 pltdune002 pltdune      62376 Aug  3 12:35 Pandora_Geometry.xml
-rw-r--r-- 1 pltdune002 pltdune      36864 Aug  3 12:35 time.db
-rw-r--r-- 1 pltdune002 pltdune        519 Aug  3 11:13 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 pltdune002 pltdune        519 Aug  3 09:07 g4_protoDUNE_hist.root
-rw-r--r-- 1 pltdune002 pltdune        519 Aug  3 12:35 protoDUNE_reco1_hist.root
-rw-r--r-- 1 pltdune002 pltdune         52 Aug  3 08:31 all-input-dids.txt
-rw-r--r-- 1 pltdune002 pltdune          7 Aug  3 12:35 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune002 pltdune          0 Aug  3 11:22 ProtoDUNE.log
-rw-r--r-- 1 pltdune002 pltdune          0 Aug  3 11:13 debugprod.log
justIN time: 2025-08-04 21:36:16 UTC       justIN version: 01.04.00