justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 295948.1@dunegpschedd02.fnal.gov

Jobsub ID295948.1@dunegpschedd02.fnal.gov
Workflow ID12448
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-27 14:23:30
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-01-27 17:51:40
From worker nodeHostnamewn1205240.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-27 14:24:58
Input filesmonte-carlo-012448-000001
JobscriptExit code0
Real time3h (12298s)
CPU time3h (11572s = 94%)
Max RSS bytes4064878592 (3876 MiB)
Outputting started2026-01-27 17:49:57
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12448/1/001/reco_stage1.root
Finished2026-01-27 17:51:40
Saved logsjustin-logs:295948.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 120153 traces, 0 summary
Retagger: tagging trace set: gauss with 90724 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 90724 traces tagged "gauss"
FrameSaver: q=3.11425e+06 n=1585538 tag=gauss
wclsFrameSaver: saving 120153 traces tagged "wiener"
FrameSaver: q=3.30419e+06 n=1399102 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 37243 XUs and 50473 XVs -> 923 XUVs
C:0 T:1 68012 XUs and 80608 XVs -> 6529 XUVs
C:0 T:2 83503 XUs and 138159 XVs -> 6949 XUVs
C:0 T:3 40707 XUs and 54811 XVs -> 422 XUVs
C:0 T:4 11648 XUs and 13245 XVs -> 243 XUVs
C:0 T:5 37319 XUs and 41017 XVs -> 3456 XUVs
C:0 T:6 63667 XUs and 69503 XVs -> 4594 XUVs
C:0 T:7 22620 XUs and 27405 XVs -> 256 XUVs
C:0 T:8 5063 XUs and 4509 XVs -> 70 XUVs
C:0 T:9 25102 XUs and 28711 XVs -> 1771 XUVs
C:0 T:10 51573 XUs and 31966 XVs -> 5965 XUVs
C:0 T:11 330 XUs and 156 XVs -> 41 XUVs
31219 XUVs total
10237 collection wire objects
28630 potential space points
Neighbour search...
2387152 tests to find 1188504 neighbours
Iterating with no regularization...
Begin: 5.78347e+07
0 4.57255e+07
1 4.49337e+07
2 4.47985e+07
3 4.47585e+07
Now with regularization...
Begin: 3.60811e+07
0 3.60468e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411999241
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411999241
fLifetime: 30000 [us]
27-Jan-2026 17:49:54 GMT  Closed output file "reco_stage1.root"
27-Jan-2026 17:49:54 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          381.925       549.468       747.707       547.64        101.047        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00133411    0.00445578    0.00652835    0.00452428    0.00166291       10     
reco:ophit:OpHitFinder                             0.320317      0.400776      0.479922      0.394953      0.0431242       10     
reco:opflash:OpFlashFinder                         0.134035       0.21727      0.302965      0.209504      0.0431249       10     
reco:opslicer:OpSlicer                              0.83105       1.05163       1.40875       1.03188      0.164683        10     
reco:caldata:DataPrepModule                         6.7451        8.10503       8.97482       8.49723      0.780154        10     
reco:wclsdatasp:WireCellToolkit                     41.754        53.585        58.3092       55.5817       5.48052        10     
reco:gaushit:GausHitFinder                          1.18081       1.70851       2.3821        1.69725      0.329774        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000577911   0.000762716   0.000918815   0.000775919   0.000103853      10     
reco:reco3d:SpacePointSolver                        11.4135       22.1625       30.3411       22.8516       6.67519        10     
reco:hitpdune:DisambigFromSpacePoints               1.30331       2.02238       3.11817       1.95784       0.54812        10     
reco:pandora:StandardPandora                        18.8245       46.3914       95.2446       44.7685       20.029         10     
reco:pandoraWriter:StandardPandora                 0.453357      0.614797      0.776344      0.626747      0.106872        10     
reco:pandoraTrack:LArPandoraTrackCreation           3.71261       5.8705        8.10977       6.27863       1.32706        10     
reco:pandoraShower:LArPandoraShowerCreation         2.58692       6.06418       9.7262        5.91908       2.27893        10     
reco:pandoracalo:Calorimetry                        1.89001       3.13376       4.59284       3.28762      0.854493        10     
reco:pandoracalonosce:Calorimetry                   1.69852       2.96744       4.35028       2.9763       0.914987        10     
reco:pandorapid:Chi2ParticleID                    0.00617264    0.00947295     0.0128629    0.00971109    0.00186593       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0498765     0.086764      0.113033      0.089118      0.0187492       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00618319    0.00920791     0.0111684    0.00972465    0.00148565       10     
reco:pandoraShowercalo:ShowerCalorimetry            2.5073        6.02765       11.0721       5.21242       2.62845        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.37546       5.66611        10.51        4.99297       2.48915        10     
reco:emtrkmichelid:EmTrackMichelId                  239.257       318.343       410.53        313.796       48.6743        10     
reco:crttag:SingleCRTMatchingProducer              0.0123136     0.0188929     0.034754      0.0176467    0.00578156       10     
reco:crtreco:TwoCRTMatchingProducer               0.00964211     0.0140296     0.0210667     0.0127013    0.00348393       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.802099       1.23844       1.50635       1.2743       0.211679        10     
reco:pandora2Track:LArPandoraTrackCreation          8.34746       19.5208       34.9311       17.5222       7.75547        10     
reco:pandora2calo:Calorimetry                       4.2188        9.28221       16.3343       8.08167       3.72065        10     
reco:pandora2calonosce:Calorimetry                  4.05253       8.58027       14.6782       7.55553       3.38366        10     
reco:pandora2pid:Chi2ParticleID                    0.0148989     0.0236503     0.0318813     0.0236499    0.00494553       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0748759     0.115082      0.155148      0.109599      0.0228289       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.014848      0.0238522     0.0320767     0.0239064    0.00495067       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.26996       7.20473       12.4045       6.81286       2.7112         10     
reco:pandora2Showercalo:ShowerCalorimetry           3.77987       7.70033       13.562        6.72041       3.00271        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.73273       6.80139       12.1124       6.14628       2.87951        10     
[art]:TriggerResults:TriggerResultInserter         3.207e-05    5.7283e-05    0.000144864    5.023e-05    3.01803e-05      10     
end_path:out1:RootOutput                           5.721e-06    8.1733e-06    1.3405e-05    7.5845e-06    2.06364e-06      10     
end_path:out1:RootOutput(write)                     2.53987       4.35466       5.98257       4.35064      0.910998        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4895.2 MB
  Peak resident set size usage (VmHWM): 4064.88 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.33 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.17 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.20 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9008836
-rw-r--r-- 1 dune01 dune01 2161342279 Jan 27 16:15 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2053834150 Jan 27 16:12 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1807931332 Jan 27 17:49 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1604170086 Jan 27 14:44 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1531423050 Jan 27 14:32 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   65713690 Jan 27 17:42 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242939 Jan 27 17:49 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Jan 27 17:49 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Jan 27 17:49 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Jan 27 17:49 time.db
-rw-r--r-- 1 dune01 dune01        519 Jan 27 16:15 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 27 14:44 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 27 17:49 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Jan 27 14:24 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Jan 27 17:49 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Jan 27 16:24 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Jan 27 16:15 debugprod.log
justIN time: 2026-02-04 18:07:56 UTC       justIN version: 01.06.00