justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 265386.0@dunegpschedd01.fnal.gov

Jobsub ID265386.0@dunegpschedd01.fnal.gov
Workflow ID11123
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-09 13:45:51
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-12-09 21:23:01
From worker nodeHostnamewn2210250.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-09 17:15:44
Input filesmonte-carlo-011123-000001
JobscriptExit code0
Real time4h (14749s)
CPU time3h (14278s = 96%)
Max RSS bytes4149104640 (3956 MiB)
Outputting started2025-12-09 21:21:34
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11123/1/001/reco_stage1.root
Finished2025-12-09 21:23:01
Saved logsjustin-logs:265386.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

BasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 122813 traces, 0 summary
Retagger: tagging trace set: gauss with 93283 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 93283 traces tagged "gauss"
FrameSaver: q=3.29257e+06 n=1663764 tag=gauss
wclsFrameSaver: saving 122813 traces tagged "wiener"
FrameSaver: q=3.50343e+06 n=1466749 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 23517 XUs and 16281 XVs -> 353 XUVs
C:0 T:1 91938 XUs and 83916 XVs -> 12523 XUVs
C:0 T:2 104974 XUs and 316280 XVs -> 17996 XUVs
C:0 T:4 9913 XUs and 12698 XVs -> 136 XUVs
C:0 T:5 52768 XUs and 119591 XVs -> 9504 XUVs
C:0 T:6 123451 XUs and 145649 XVs -> 7719 XUVs
C:0 T:7 46385 XUs and 43769 XVs -> 433 XUVs
C:0 T:8 12203 XUs and 11469 XVs -> 2079 XUVs
C:0 T:9 67995 XUs and 60920 XVs -> 5584 XUVs
C:0 T:10 40954 XUs and 23302 XVs -> 3949 XUVs
C:0 T:11 235 XUs and 155 XVs -> 20 XUVs
60296 XUVs total
12444 collection wire objects
55571 potential space points
Neighbour search...
10566039 tests to find 4454076 neighbours
Iterating with no regularization...
Begin: 7.91102e+07
0 5.12708e+07
1 5.01862e+07
2 4.99751e+07
3 4.98948e+07
4 4.98535e+07
Now with regularization...
Begin: 4.02155e+07
0 4.01399e+07
1 4.00854e+07
2 4.00426e+07
3 4.00065e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411012837
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411012837
fLifetime: 30000 [us]
09-Dec-2025 21:21:31 GMT  Closed output file "reco_stage1.root"
09-Dec-2025 21:21:31 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          321.371       473.674       561.277       508.093       80.9126        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00182276    0.00624306     0.0112285    0.00577707    0.00285047       10     
reco:ophit:OpHitFinder                             0.276332      0.386395      0.484427      0.398353      0.0766115       10     
reco:opflash:OpFlashFinder                          0.14031      0.233493      0.347835      0.228009      0.0639026       10     
reco:opslicer:OpSlicer                             0.509751      0.777421       1.0704       0.718816      0.195857        10     
reco:caldata:DataPrepModule                         5.67805       7.53313       9.69809       7.62781       1.06777        10     
reco:wclsdatasp:WireCellToolkit                     40.1073       52.5064       65.9515       54.5028       7.33902        10     
reco:gaushit:GausHitFinder                         0.965134       1.8274        2.22053       1.9636       0.376871        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000427362   0.00130857    0.00329502    0.00098276    0.000821089      10     
reco:reco3d:SpacePointSolver                        9.6316        21.9454       29.0995       23.6588       6.04131        10     
reco:hitpdune:DisambigFromSpacePoints               1.37698       2.2648        2.81764       2.4397       0.475614        10     
reco:pandora:StandardPandora                        24.4923       42.2882       64.7749       40.2687       12.8785        10     
reco:pandoraWriter:StandardPandora                 0.464218      0.690166       1.21654      0.670896      0.207352        10     
reco:pandoraTrack:LArPandoraTrackCreation           3.45818       7.12979       15.3382       6.76256       3.09893        10     
reco:pandoraShower:LArPandoraShowerCreation         3.07942       8.32526       18.7354       8.19052       4.18775        10     
reco:pandoracalo:Calorimetry                        2.17709       4.15332       9.21715       3.59032       1.93783        10     
reco:pandoracalonosce:Calorimetry                   1.92312       4.02373       8.80706       3.78857       1.86588        10     
reco:pandorapid:Chi2ParticleID                    0.00405962    0.00893077     0.0191179    0.00862603    0.00388561       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0410346     0.0753864     0.0993661     0.0832589     0.0199328       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.0052849    0.00794252     0.0102851    0.00773624    0.00157787       10     
reco:pandoraShowercalo:ShowerCalorimetry            2.82289       8.80602       20.1244       8.28635       4.77116        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.8137        8.19216       16.5448       7.79321       3.72467        10     
reco:emtrkmichelid:EmTrackMichelId                  179.59        234.873       281.013       239.498       29.7632        10     
reco:crttag:SingleCRTMatchingProducer              0.0135388     0.0224379     0.0398837     0.0207804    0.00718434       10     
reco:crtreco:TwoCRTMatchingProducer               0.00613993     0.0133546     0.0206756     0.0132855    0.00382668       10     
reco:anodepiercerst0:T0RecoAnodePiercers            0.74259       1.1099        1.73161       1.10931      0.244113        10     
reco:pandora2Track:LArPandoraTrackCreation          8.47308       19.9928       30.7126       21.1237       7.55987        10     
reco:pandora2calo:Calorimetry                       3.72657       9.97655       16.1687       10.2524       3.72586        10     
reco:pandora2calonosce:Calorimetry                  3.90113       9.70474       15.1412       10.1579       3.94536        10     
reco:pandora2pid:Chi2ParticleID                   0.00876685     0.0183873     0.0271848     0.0204465    0.00607475       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0480432     0.090852      0.126064      0.100823      0.0280455       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.011655      0.0183374     0.0252733     0.0199612    0.00488823       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.55651       7.12058        10.57        7.2498        2.25967        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.42888       8.24455       13.9493       8.11534       2.96377        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.03641       7.66982       13.6309       8.19492       2.74756        10     
[art]:TriggerResults:TriggerResultInserter        6.1936e-05    0.000115635   0.000292058   0.00010393    6.18459e-05      10     
end_path:out1:RootOutput                           9.979e-06    1.95097e-05   4.7299e-05     1.595e-05    1.05645e-05      10     
end_path:out1:RootOutput(write)                     2.61264       3.41245       4.58171        3.178       0.688659        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5019.76 MB
  Peak resident set size usage (VmHWM): 4149.1 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.14 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.37 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.59 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.19 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10258664
-rw-r--r-- 1 dune01 dune01 2410583004 Dec  9 20:00 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2289620793 Dec  9 19:56 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2089334865 Dec  9 21:21 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1855922576 Dec  9 17:37 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1785721843 Dec  9 17:23 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   73055530 Dec  9 21:16 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     243130 Dec  9 21:21 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Dec  9 21:21 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Dec  9 21:21 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Dec  9 21:21 time.db
-rw-r--r-- 1 dune01 dune01        519 Dec  9 20:00 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec  9 17:37 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec  9 21:21 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Dec  9 17:15 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Dec  9 21:21 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Dec  9 20:09 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Dec  9 20:00 debugprod.log
justIN time: 2026-02-07 15:00:10 UTC       justIN version: 01.06.00