justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 271829.0@dunegpschedd02.fnal.gov

Jobsub ID271829.0@dunegpschedd02.fnal.gov
Workflow ID11879
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-14 06:32:15
SiteCZ_FZU
EntryNova_CZ_FZU_osgce2
Last heartbeat2026-01-14 11:58:04
From worker nodeHostnamemahagon22.farm.particle.cz
cpuinfoAMD EPYC 7301 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-14 06:33:08
Input filesmonte-carlo-011879-000001
JobscriptExit code0
Real time5h (19447s)
CPU time5h (18823s = 96%)
Max RSS bytes4359340032 (4157 MiB)
Outputting started2026-01-14 11:57:16
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11879/1/001/reco_stage1.root
Finished2026-01-14 11:58:04
Saved logsjustin-logs:271829.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 106613 traces, 0 summary
Retagger: tagging trace set: gauss with 88824 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 88824 traces tagged "gauss"
FrameSaver: q=3.2438e+06 n=1733364 tag=gauss
wclsFrameSaver: saving 106613 traces tagged "wiener"
FrameSaver: q=3.38071e+06 n=1589250 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 38759 XUs and 45651 XVs -> 515 XUVs
C:0 T:1 69300 XUs and 88280 XVs -> 5955 XUVs
C:0 T:2 39763 XUs and 62738 XVs -> 2535 XUVs
C:0 T:3 24236 XUs and 29012 XVs -> 190 XUVs
C:0 T:4 16482 XUs and 14121 XVs -> 92 XUVs
C:0 T:5 48020 XUs and 39868 XVs -> 3754 XUVs
C:0 T:6 47340 XUs and 44467 XVs -> 1813 XUVs
C:0 T:7 22171 XUs and 19622 XVs -> 149 XUVs
C:0 T:8 13061 XUs and 10561 XVs -> 137 XUVs
C:0 T:9 82548 XUs and 93201 XVs -> 3883 XUVs
C:0 T:10 38640 XUs and 15334 XVs -> 3083 XUVs
C:0 T:11 236 XUs and 160 XVs -> 21 XUVs
22127 XUVs total
9162 collection wire objects
20392 potential space points
Neighbour search...
1043324 tests to find 548810 neighbours
Iterating with no regularization...
Begin: 7.55952e+07
0 6.83519e+07
1 6.79142e+07
2 6.78555e+07
Now with regularization...
Begin: 5.91883e+07
0 5.91682e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411730985
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411730985
fLifetime: 30000 [us]
14-Jan-2026 12:57:12 CET  Closed output file "reco_stage1.root"
14-Jan-2026 12:57:12 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          601.048       701.407       835.696       662.097       83.2756        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00187611    0.00605266    0.00960737    0.00624511    0.00237418       10     
reco:ophit:OpHitFinder                             0.438202      0.502512      0.568446      0.510512      0.0489574       10     
reco:opflash:OpFlashFinder                         0.272579      0.335766      0.419909      0.317946      0.0446698       10     
reco:opslicer:OpSlicer                             0.875376       1.32749       1.67819       1.32125      0.225272        10     
reco:caldata:DataPrepModule                         7.97654       9.19266       10.6323       9.1448        0.72397        10     
reco:wclsdatasp:WireCellToolkit                     54.9485       63.7549       73.414        63.9866       5.55209        10     
reco:gaushit:GausHitFinder                          1.86967       2.27425       2.77638       2.23154      0.258574        10     
reco:nhitsfilter:NumberOfHitsFilter                0.0007511    0.000995748   0.00135074    0.000850177   0.000236226      10     
reco:reco3d:SpacePointSolver                        14.6872       24.0754       41.8123       22.7399       7.1899         10     
reco:hitpdune:DisambigFromSpacePoints               2.34583       3.14921       4.35192       3.08707      0.556764        10     
reco:pandora:StandardPandora                        48.6235       78.4604        137.6        68.8858       25.7578        10     
reco:pandoraWriter:StandardPandora                 0.640546      0.844953       1.02635       0.86634      0.118385        10     
reco:pandoraTrack:LArPandoraTrackCreation           8.77717       10.8842       13.3512       10.8536       1.60348        10     
reco:pandoraShower:LArPandoraShowerCreation         7.56021       11.1856       17.9192       10.6483       2.95503        10     
reco:pandoracalo:Calorimetry                        3.96217       5.63256       8.76625       5.07292       1.40764        10     
reco:pandoracalonosce:Calorimetry                   3.91289       5.44703       8.23319       5.06972       1.27239        10     
reco:pandorapid:Chi2ParticleID                    0.00867053     0.0128401     0.0169823     0.0127223    0.00265917       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0636214      0.10203      0.130722      0.0991726     0.0215025       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00886232     0.0111231     0.013856      0.0111627    0.00132864       10     
reco:pandoraShowercalo:ShowerCalorimetry            7.56216       11.2737       18.7354       10.9839       3.10391        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       7.12784       11.4628       17.1692       11.4497       3.19835        10     
reco:emtrkmichelid:EmTrackMichelId                  314.325       366.594       438.595       354.724       35.3905        10     
reco:crttag:SingleCRTMatchingProducer              0.0226023     0.0336187     0.0424129     0.0354561    0.00636309       10     
reco:crtreco:TwoCRTMatchingProducer                0.0153081     0.0204812     0.0351189     0.0190159    0.00536224       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.11849       1.53374       1.92647       1.5867       0.213602        10     
reco:pandora2Track:LArPandoraTrackCreation          21.1396       27.9194       37.3861       26.199        5.57396        10     
reco:pandora2calo:Calorimetry                       10.7466       14.1124       17.8172       13.3243       2.65882        10     
reco:pandora2calonosce:Calorimetry                  9.9865        12.924        16.3202       12.6079       2.20264        10     
reco:pandora2pid:Chi2ParticleID                    0.0240421     0.0288331     0.0349488     0.028797     0.00337489       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0808748      0.13089       0.15805      0.130667      0.0202662       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0161508     0.0270992     0.0329743     0.0275966    0.00452238       10     
reco:pandora2Shower:LArPandoraShowerCreation        8.11539       10.0385       11.8503       10.0011       1.2361         10     
reco:pandora2Showercalo:ShowerCalorimetry           9.37646       12.2276       14.6416       12.8892       1.80884        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      7.25706       10.4328       14.2059       10.0199       1.95025        10     
[art]:TriggerResults:TriggerResultInserter         8.436e-05    0.000134922   0.000277615   0.000117031   5.59883e-05      10     
end_path:out1:RootOutput                          1.3386e-05    1.92105e-05   3.5888e-05    1.56595e-05   6.79112e-06      10     
end_path:out1:RootOutput(write)                     4.16738       5.22285       6.37863       5.19841       0.67157        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5184.14 MB
  Peak resident set size usage (VmHWM): 4359.34 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.20 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.39 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.52 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.22 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.13 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11102844
-rw-r--r-- 1 fermilab users 2570291480 Jan 14 10:56 detsim_stage2.root
-rw-r--r-- 1 fermilab users 2435262331 Jan 14 10:51 detsim_stage1.root
-rw-r--r-- 1 fermilab users 2275288142 Jan 14 12:57 reco_stage1.root
-rw-r--r-- 1 fermilab users 2046770392 Jan 14 08:01 g4_stage2.root
-rw-r--r-- 1 fermilab users 1961493741 Jan 14 07:42 g4_stage1.root
-rw-r--r-- 1 fermilab users   79576770 Jan 14 12:49 Pandora_Events.pndr
-rw-r--r-- 1 fermilab users     242903 Jan 14 12:57 jobscript.log
-rw-r--r-- 1 fermilab users     217088 Jan 14 12:57 mem.db
-rw-r--r-- 1 fermilab users      62376 Jan 14 12:57 Pandora_Geometry.xml
-rw-r--r-- 1 fermilab users      36864 Jan 14 12:57 time.db
-rw-r--r-- 1 fermilab users        519 Jan 14 10:56 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 fermilab users        519 Jan 14 08:01 g4_protoDUNE_hist.root
-rw-r--r-- 1 fermilab users        519 Jan 14 12:57 protoDUNE_reco1_hist.root
-rw-r--r-- 1 fermilab users         52 Jan 14 07:33 all-input-dids.txt
-rw-r--r-- 1 fermilab users          7 Jan 14 12:57 justin-processed-pfns.txt
-rw-r--r-- 1 fermilab users          0 Jan 14 11:08 ProtoDUNE.log
-rw-r--r-- 1 fermilab users          0 Jan 14 10:56 debugprod.log
justIN time: 2026-02-06 04:51:39 UTC       justIN version: 01.06.00