justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 236979.0@dunegpschedd02.fnal.gov

Jobsub ID236979.0@dunegpschedd02.fnal.gov
Workflow ID9571
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-03 21:47:18
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-11-04 04:24:53
From worker nodeHostnamewn1208081.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-03 21:49:20
Input filesmonte-carlo-009571-000001
JobscriptExit code0
Real time6h (23614s)
CPU time6h (22744s = 96%)
Max RSS bytes4474515456 (4267 MiB)
Outputting started2025-11-04 04:22:55
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09571/1/001/reco_stage1.root
Finished2025-11-04 04:24:53
Saved logsjustin-logs:236979.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

AllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 118173 traces, 0 summary
Retagger: tagging trace set: gauss with 90407 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 90407 traces tagged "gauss"
FrameSaver: q=3.0143e+06 n=1579168 tag=gauss
wclsFrameSaver: saving 118173 traces tagged "wiener"
FrameSaver: q=3.20037e+06 n=1395255 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 31000 XUs and 28018 XVs -> 678 XUVs
C:0 T:1 65091 XUs and 54382 XVs -> 6814 XUVs
C:0 T:2 115466 XUs and 140880 XVs -> 6875 XUVs
C:0 T:3 55164 XUs and 63777 XVs -> 659 XUVs
C:0 T:4 6796 XUs and 4964 XVs -> 82 XUVs
C:0 T:5 28457 XUs and 13867 XVs -> 3637 XUVs
C:0 T:6 91301 XUs and 76333 XVs -> 3239 XUVs
C:0 T:7 43695 XUs and 31450 XVs -> 263 XUVs
C:0 T:8 7108 XUs and 6610 XVs -> 55 XUVs
C:0 T:9 74971 XUs and 51394 XVs -> 5368 XUVs
C:0 T:10 42794 XUs and 26755 XVs -> 3300 XUVs
C:0 T:11 167 XUs and 103 XVs -> 17 XUVs
30987 XUVs total
10526 collection wire objects
28656 potential space points
Neighbour search...
2258658 tests to find 1202536 neighbours
Iterating with no regularization...
Begin: 5.97057e+07
0 4.57743e+07
1 4.43567e+07
2 4.42205e+07
3 4.41869e+07
Now with regularization...
Begin: 3.53598e+07
0 3.53179e+07
1 3.52949e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410295615
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410295615
fLifetime: 30000 [us]
04-Nov-2025 04:22:52 GMT  Closed output file "reco_stage1.root"
04-Nov-2025 04:22:52 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          643.931       874.445       1171.67       899.356       156.654        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00222994    0.00871631     0.0189815     0.0072837    0.00548391       10     
reco:ophit:OpHitFinder                             0.427006      0.522703      0.684392      0.496435      0.0747897       10     
reco:opflash:OpFlashFinder                         0.248166      0.348563      0.451902      0.351658      0.0658366       10     
reco:opslicer:OpSlicer                              1.21419       1.3846        1.56814       1.32924      0.121293        10     
reco:caldata:DataPrepModule                         7.95826       8.93155       9.75205       8.90981      0.486846        10     
reco:wclsdatasp:WireCellToolkit                     54.3113       61.3206       68.9709       60.6752       4.30844        10     
reco:gaushit:GausHitFinder                          1.90374       2.47158       3.07081       2.50782      0.366748        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00107435    0.00137251    0.00190686     0.0012546    0.000258546      10     
reco:reco3d:SpacePointSolver                        17.949        36.4543       52.7206       37.291        11.6725        10     
reco:hitpdune:DisambigFromSpacePoints               2.00481       3.70407       5.19976       3.76133      0.962771        10     
reco:pandora:StandardPandora                        40.724        82.9227       131.745       80.1325       28.5666        10     
reco:pandoraWriter:StandardPandora                  0.75741      0.966035       1.21359      0.974225      0.135167        10     
reco:pandoraTrack:LArPandoraTrackCreation           9.03504       13.9969       16.9962       14.6299       2.50744        10     
reco:pandoraShower:LArPandoraShowerCreation         9.91491       16.0553       23.3412       15.176        4.66718        10     
reco:pandoracalo:Calorimetry                        6.08819       8.53446       10.6478       8.91711       1.76519        10     
reco:pandoracalonosce:Calorimetry                   5.55542       8.43876       11.0347       9.02856       1.91875        10     
reco:pandorapid:Chi2ParticleID                     0.0101707     0.0123097     0.0173121     0.0121524    0.00204256       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0955607     0.133288       0.23409      0.128471      0.0373398       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.0100012     0.0125816     0.0187316     0.011511     0.00243698       10     
reco:pandoraShowercalo:ShowerCalorimetry            11.8629       18.8117       29.3454       17.4338       5.89251        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       11.0082       18.2742       29.5474       17.7706       5.25735        10     
reco:emtrkmichelid:EmTrackMichelId                  344.266       437.701       564.369       446.129       66.5825        10     
reco:crttag:SingleCRTMatchingProducer              0.0276547     0.0443956     0.070018      0.0402058     0.0112042       10     
reco:crtreco:TwoCRTMatchingProducer                0.0219822     0.0302711     0.0366685     0.0303451    0.00375089       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.06944       1.66934       2.06688       1.73778      0.267731        10     
reco:pandora2Track:LArPandoraTrackCreation          28.367        44.6659       64.5063       44.3929       10.6792        10     
reco:pandora2calo:Calorimetry                       16.5391       24.1839       34.5199       23.6534       5.85311        10     
reco:pandora2calonosce:Calorimetry                  12.4361       22.5436       29.9302       22.0489        5.307         10     
reco:pandora2pid:Chi2ParticleID                    0.0234822     0.0303408     0.0384694     0.0301864    0.00531262       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.129892      0.158978       0.19285      0.156626      0.0195677       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0212301     0.0289006     0.0382318     0.0297596     0.0049191       10     
reco:pandora2Shower:LArPandoraShowerCreation         11.4         15.2423       20.3536       15.4986       2.87289        10     
reco:pandora2Showercalo:ShowerCalorimetry           15.2863       20.5314       32.9108       18.7784       5.32392        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      12.6091       18.2557       25.3101       17.4431       3.64446        10     
[art]:TriggerResults:TriggerResultInserter        4.6658e-05    6.86204e-05   0.000162908   5.26895e-05   3.42546e-05      10     
end_path:out1:RootOutput                           6.863e-06    8.6201e-06    1.5479e-05    7.9845e-06    2.37197e-06      10     
end_path:out1:RootOutput(write)                     4.43372       5.75167       7.23392       5.75979      0.874522        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5340.43 MB
  Peak resident set size usage (VmHWM): 4474.52 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.36 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.73 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.22 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11200528
-rw-r--r-- 1 dune01 dune01 2596947923 Nov  4 01:53 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2460853082 Nov  4 01:48 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2333995196 Nov  4 04:22 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 2036826281 Nov  3 22:25 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1953353590 Nov  3 21:59 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   86729520 Nov  4 04:13 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     243424 Nov  4 04:22 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Nov  4 04:22 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Nov  4 04:22 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Nov  4 04:22 time.db
-rw-r--r-- 1 dune01 dune01        519 Nov  4 01:53 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov  3 22:25 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov  4 04:22 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Nov  3 21:49 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Nov  4 04:22 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Nov  4 02:08 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Nov  4 01:53 debugprod.log
justIN time: 2026-02-11 12:00:21 UTC       justIN version: 01.06.00