justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 264370.0@dunegpschedd01.fnal.gov

Jobsub ID264370.0@dunegpschedd01.fnal.gov
Workflow ID11059
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-06 11:33:57
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-12-06 15:48:36
From worker nodeHostnamewn2208250.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-06 11:35:12
Input filesmonte-carlo-011059-000001
JobscriptExit code0
Real time4h (15108s)
CPU time3h (14362s = 95%)
Max RSS bytes4371402752 (4168 MiB)
Outputting started2025-12-06 15:47:01
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11059/1/001/reco_stage1.root
Finished2025-12-06 15:48:36
Saved logsjustin-logs:264370.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 95580 traces, 0 summary
Retagger: tagging trace set: gauss with 72218 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 72218 traces tagged "gauss"
FrameSaver: q=2.55468e+06 n=1273010 tag=gauss
wclsFrameSaver: saving 95580 traces tagged "wiener"
FrameSaver: q=2.70777e+06 n=1119728 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 21831 XUs and 19394 XVs -> 87 XUVs
C:0 T:1 39590 XUs and 33751 XVs -> 3328 XUVs
C:0 T:2 63062 XUs and 88547 XVs -> 4633 XUVs
C:0 T:3 27853 XUs and 35045 XVs -> 149 XUVs
C:0 T:4 6362 XUs and 5584 XVs -> 30 XUVs
C:0 T:5 18261 XUs and 11013 XVs -> 1363 XUVs
C:0 T:6 190939 XUs and 103100 XVs -> 9287 XUVs
C:0 T:7 43350 XUs and 33560 XVs -> 389 XUVs
C:0 T:8 6018 XUs and 6903 XVs -> 93 XUVs
C:0 T:9 34513 XUs and 66839 XVs -> 3570 XUVs
C:0 T:10 82312 XUs and 15127 XVs -> 6571 XUVs
C:0 T:11 49 XUs and 30 XVs -> 3 XUVs
29503 XUVs total
9144 collection wire objects
27419 potential space points
Neighbour search...
3245593 tests to find 1550786 neighbours
Iterating with no regularization...
Begin: 5.80264e+07
0 4.35888e+07
1 4.28315e+07
2 4.27076e+07
3 4.26676e+07
Now with regularization...
Begin: 3.50662e+07
0 3.50186e+07
1 3.49924e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410949927
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410949927
fLifetime: 30000 [us]
06-Dec-2025 15:46:59 GMT  Closed output file "reco_stage1.root"
06-Dec-2025 15:46:59 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          314.377       483.855       622.815       494.202       80.1311        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00140145     0.0050025     0.0085412    0.00442619    0.00220466       10     
reco:ophit:OpHitFinder                             0.306148      0.404117      0.481473      0.407987      0.0493944       10     
reco:opflash:OpFlashFinder                         0.151172      0.221947       0.26573      0.226808      0.0370835       10     
reco:opslicer:OpSlicer                             0.602734      0.860809      0.997879      0.874089      0.108791        10     
reco:caldata:DataPrepModule                         6.8977        7.50724       8.31311       7.41899      0.401774        10     
reco:wclsdatasp:WireCellToolkit                     40.8119       46.0898       53.7538       44.4314       4.22135        10     
reco:gaushit:GausHitFinder                          1.1828        1.70435       2.27413       1.77828      0.310284        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000458792   0.000796588   0.00139192    0.000793054   0.000263651      10     
reco:reco3d:SpacePointSolver                        13.376        26.8617       34.5682       29.2407       6.60551        10     
reco:hitpdune:DisambigFromSpacePoints               1.23679       2.42673       3.13608       2.50281      0.532085        10     
reco:pandora:StandardPandora                        19.7555       48.3769       87.5978       46.9578       17.1951        10     
reco:pandoraWriter:StandardPandora                 0.449742      0.665981      0.961278      0.674153      0.140583        10     
reco:pandoraTrack:LArPandoraTrackCreation           2.69592       5.95684       7.8214        6.26761       1.36571        10     
reco:pandoraShower:LArPandoraShowerCreation         2.96286       7.04989       11.271        7.38902       2.18751        10     
reco:pandoracalo:Calorimetry                        1.76214       3.66885       5.28614       3.45065       0.94797        10     
reco:pandoracalonosce:Calorimetry                   1.5712        3.35182       4.40881       3.42397       0.74302        10     
reco:pandorapid:Chi2ParticleID                    0.00460008    0.00761237    0.00970748     0.0087422    0.00208403       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0485338     0.0730309     0.109747      0.0683188     0.0195715       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00474564    0.00719918     0.010651     0.00737214    0.00173617       10     
reco:pandoraShowercalo:ShowerCalorimetry            3.28148       7.48388       9.53479       7.9673        1.76093        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.09525       6.88304       9.30635       7.05922       1.74387        10     
reco:emtrkmichelid:EmTrackMichelId                  178.403       245.238       300.881       252.801       34.5052        10     
reco:crttag:SingleCRTMatchingProducer              0.0148415     0.0232155     0.0325431     0.0227377    0.00486619       10     
reco:crtreco:TwoCRTMatchingProducer               0.00867136     0.0148354     0.0192358     0.0149522    0.00351882       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.753581       1.16911       1.36822       1.20673      0.173803        10     
reco:pandora2Track:LArPandoraTrackCreation          9.01115       20.0153       26.6704       20.488        5.15186        10     
reco:pandora2calo:Calorimetry                       4.33005       10.2447        13.38        10.2326       2.66775        10     
reco:pandora2calonosce:Calorimetry                  4.11771       9.63943       13.2523       9.23294       2.58553        10     
reco:pandora2pid:Chi2ParticleID                    0.0121902     0.0204784     0.0271372     0.0212627    0.00382181       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0511672     0.0973127      0.11736      0.106621      0.0218514       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00939257     0.0177459     0.0231752     0.019155      0.0043072       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.34592       7.15552       9.62729       7.07343       1.77515        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.79667       8.65151       11.6771       8.38207       2.33728        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.26347       7.82868       10.7643       7.78346       2.02407        10     
[art]:TriggerResults:TriggerResultInserter        8.9007e-05    0.000111086   0.000194225   0.000100418   2.99493e-05      10     
end_path:out1:RootOutput                          1.2364e-05    1.56334e-05   2.9505e-05    1.44275e-05   4.71503e-06      10     
end_path:out1:RootOutput(write)                     2.72803       3.95991       4.73336       3.96884      0.580091        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5227.11 MB
  Peak resident set size usage (VmHWM): 4371.4 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.14 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.35 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.54 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.19 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10868788
-rw-r--r-- 1 dune01 dune01 2524420375 Dec  6 14:23 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2397799916 Dec  6 14:20 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2240567164 Dec  6 15:46 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1982719832 Dec  6 12:03 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1901454310 Dec  6 11:43 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   82046050 Dec  6 15:43 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     243066 Dec  6 15:47 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Dec  6 15:46 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Dec  6 15:46 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Dec  6 15:46 time.db
-rw-r--r-- 1 dune01 dune01        519 Dec  6 14:23 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec  6 12:03 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec  6 15:46 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Dec  6 11:35 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Dec  6 15:47 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Dec  6 14:33 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Dec  6 14:23 debugprod.log
justIN time: 2026-02-07 17:42:23 UTC       justIN version: 01.06.00