justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 248110.1@dunegpschedd02.fnal.gov

Jobsub ID248110.1@dunegpschedd02.fnal.gov
Workflow ID10332
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-18 15:27:52
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-11-18 20:24:27
From worker nodeHostnamewn1905251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-18 15:29:10
Input filesmonte-carlo-010332-000001
JobscriptExit code0
Real time4h (17620s)
CPU time4h (17095s = 97%)
Max RSS bytes4384808960 (4181 MiB)
Outputting started2025-11-18 20:22:50
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10332/1/001/reco_stage1.root
Finished2025-11-18 20:24:27
Saved logsjustin-logs:248110.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 118987 traces, 0 summary
Retagger: tagging trace set: gauss with 88812 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 88812 traces tagged "gauss"
FrameSaver: q=3.16405e+06 n=1526138 tag=gauss
wclsFrameSaver: saving 118987 traces tagged "wiener"
FrameSaver: q=3.36224e+06 n=1343399 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 21378 XUs and 26668 XVs -> 219 XUVs
C:0 T:1 40177 XUs and 44799 XVs -> 3402 XUVs
C:0 T:2 85391 XUs and 77340 XVs -> 4725 XUVs
C:0 T:3 43491 XUs and 33822 XVs -> 221 XUVs
C:0 T:4 14499 XUs and 16364 XVs -> 226 XUVs
C:0 T:5 50929 XUs and 45277 XVs -> 3587 XUVs
C:0 T:6 88873 XUs and 57134 XVs -> 4780 XUVs
C:0 T:7 39383 XUs and 22438 XVs -> 323 XUVs
C:0 T:8 5756 XUs and 7551 XVs -> 68 XUVs
C:0 T:9 36668 XUs and 63698 XVs -> 4116 XUVs
C:0 T:10 38461 XUs and 29054 XVs -> 7321 XUVs
C:0 T:11 505 XUs and 439 XVs -> 82 XUVs
29070 XUVs total
9695 collection wire objects
27211 potential space points
Neighbour search...
2138347 tests to find 1056624 neighbours
Iterating with no regularization...
Begin: 6.07122e+07
0 5.22376e+07
1 5.13465e+07
2 5.11966e+07
3 5.11508e+07
Now with regularization...
Begin: 4.16421e+07
0 4.16181e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410591958
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410591958
fLifetime: 30000 [us]
18-Nov-2025 20:22:47 GMT  Closed output file "reco_stage1.root"
18-Nov-2025 20:22:47 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          303.659       529.27        731.746       535.958       132.13         10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00182267    0.00574288     0.0108697    0.00526113    0.00267003       10     
reco:ophit:OpHitFinder                             0.233293      0.366637      0.453684      0.387248      0.0657386       10     
reco:opflash:OpFlashFinder                         0.151586      0.251541      0.368405      0.262348      0.0666436       10     
reco:opslicer:OpSlicer                             0.606859      0.857792       1.28263      0.858621      0.186998        10     
reco:caldata:DataPrepModule                         4.82636       6.51606       8.68579       6.68818       1.02358        10     
reco:wclsdatasp:WireCellToolkit                     33.9839       49.0854       61.7591       48.6574       8.77408        10     
reco:gaushit:GausHitFinder                          1.01183       1.45248       2.22699       1.26094      0.441213        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000541877   0.00137888    0.00360023    0.00119115    0.000838855      10     
reco:reco3d:SpacePointSolver                        11.0479       22.5489       42.0445       18.8017       9.4661         10     
reco:hitpdune:DisambigFromSpacePoints               1.16231       2.15787       3.80577       1.92377      0.852996        10     
reco:pandora:StandardPandora                        24.3182       53.1059       113.575       48.0984       28.9654        10     
reco:pandoraWriter:StandardPandora                 0.406741      0.806813       1.20007      0.848797      0.232493        10     
reco:pandoraTrack:LArPandoraTrackCreation           4.8222        10.0969       15.0442       9.06097       3.54452        10     
reco:pandoraShower:LArPandoraShowerCreation         4.05268       11.4268       23.0518       10.3476       5.38513        10     
reco:pandoracalo:Calorimetry                        2.62381       5.48148       8.40923       5.44166       2.02495        10     
reco:pandoracalonosce:Calorimetry                   2.48084       4.98028       7.41149       4.27987       1.74091        10     
reco:pandorapid:Chi2ParticleID                    0.00508034    0.00943945     0.0168531    0.00914005    0.00348166       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0431801     0.0924793     0.160732      0.0807414     0.0397109       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00456911     0.0095804     0.0149857    0.00932818    0.00347904       10     
reco:pandoraShowercalo:ShowerCalorimetry            4.17415       11.2664       17.2648       12.4304       4.12384        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       4.80533       10.4474       16.4015       9.4789        3.70805        10     
reco:emtrkmichelid:EmTrackMichelId                  156.626       245.39        298.488       255.114       44.8192        10     
reco:crttag:SingleCRTMatchingProducer              0.0111973     0.0303193     0.056513      0.0317964     0.0131285       10     
reco:crtreco:TwoCRTMatchingProducer               0.00815225     0.0218008     0.0407629     0.0187136     0.0109998       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.810631       1.1105        1.28995       1.17984      0.151376        10     
reco:pandora2Track:LArPandoraTrackCreation          11.5158       26.7967       42.5729       24.4513       9.18512        10     
reco:pandora2calo:Calorimetry                        6.299        14.9433       24.1563       14.3816       5.38283        10     
reco:pandora2calonosce:Calorimetry                  5.71247       14.3215       24.4894       13.7634       5.53266        10     
reco:pandora2pid:Chi2ParticleID                    0.0117535     0.0172517     0.027578      0.0157866    0.00506027       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0515626     0.081211      0.121605       0.08192      0.0215564       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.010284      0.0163966     0.022584      0.0171058    0.00407582       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.92091       9.62235       14.4778       10.1007       3.19571        10     
reco:pandora2Showercalo:ShowerCalorimetry           5.09305       11.6539       18.4415       10.8987       4.07313        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      4.7225        10.3675       15.7914       9.75453       3.47452        10     
[art]:TriggerResults:TriggerResultInserter         6.917e-05    0.00011049    0.000256792   9.41065e-05   5.1621e-05       10     
end_path:out1:RootOutput                           1.083e-05    1.6634e-05     2.661e-05    1.53585e-05   4.96392e-06      10     
end_path:out1:RootOutput(write)                     2.38931       3.64848       5.40596       3.31034      0.863619        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5216.05 MB
  Peak resident set size usage (VmHWM): 4384.81 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.10 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.33 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.91 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.15 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10820360
-rw-r--r-- 1 dune01 dune01 2524768451 Nov 18 18:52 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2396492303 Nov 18 18:48 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2226239686 Nov 18 20:22 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1982621943 Nov 18 15:54 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1870692860 Nov 18 15:37 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   78599750 Nov 18 20:15 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242966 Nov 18 20:22 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Nov 18 20:22 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Nov 18 20:22 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Nov 18 20:22 time.db
-rw-r--r-- 1 dune01 dune01        519 Nov 18 18:52 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 18 15:54 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Nov 18 20:22 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Nov 18 15:29 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Nov 18 20:22 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Nov 18 19:01 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Nov 18 18:52 debugprod.log
justIN time: 2026-02-09 06:14:12 UTC       justIN version: 01.06.00