justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 268871.0@dunegpschedd02.fnal.gov

Jobsub ID268871.0@dunegpschedd02.fnal.gov
Workflow ID11731
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-11 13:08:49
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce02
Last heartbeat2026-01-11 17:47:45
From worker nodeHostnamecn051.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-11 13:09:25
Input filesmonte-carlo-011731-000001
JobscriptExit code0
Real time4h (16478s)
CPU time4h (16228s = 98%)
Max RSS bytes4419416064 (4214 MiB)
Outputting started2026-01-11 17:44:04
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11731/1/001/reco_stage1.root
Finished2026-01-11 17:47:45
Saved logsjustin-logs:268871.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Channels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 145606 traces, 0 summary
Retagger: tagging trace set: gauss with 111794 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 111794 traces tagged "gauss"
FrameSaver: q=3.76323e+06 n=2005576 tag=gauss
wclsFrameSaver: saving 145606 traces tagged "wiener"
FrameSaver: q=3.97111e+06 n=1767375 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 33043 XUs and 37504 XVs -> 310 XUVs
C:0 T:1 66805 XUs and 68556 XVs -> 3490 XUVs
C:0 T:2 158358 XUs and 188597 XVs -> 14701 XUVs
C:0 T:3 65191 XUs and 55530 XVs -> 908 XUVs
C:0 T:4 13977 XUs and 11826 XVs -> 84 XUVs
C:0 T:5 54395 XUs and 36769 XVs -> 4503 XUVs
C:0 T:6 141716 XUs and 104757 XVs -> 7296 XUVs
C:0 T:7 51059 XUs and 33730 XVs -> 375 XUVs
C:0 T:8 8368 XUs and 6350 XVs -> 137 XUVs
C:0 T:9 56064 XUs and 56942 XVs -> 4011 XUVs
C:0 T:10 44836 XUs and 20260 XVs -> 3761 XUVs
C:0 T:11 257 XUs and 144 XVs -> 17 XUVs
39593 XUVs total
14193 collection wire objects
35438 potential space points
Neighbour search...
3281920 tests to find 1497126 neighbours
Iterating with no regularization...
Begin: 6.7509e+07
0 5.25001e+07
1 5.13253e+07
2 5.1159e+07
3 5.11128e+07
Now with regularization...
Begin: 4.05299e+07
0 4.0467e+07
1 4.0436e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411676346
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411676346
fLifetime: 30000 [us]
11-Jan-2026 17:44:01 GMT  Closed output file "reco_stage1.root"
11-Jan-2026 17:44:01 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          437.975       677.687       824.52        734.966       127.472        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00139089    0.00484589    0.00807122    0.00503111    0.00195414       10     
reco:ophit:OpHitFinder                             0.350979      0.441041      0.543682      0.448015      0.0580477       10     
reco:opflash:OpFlashFinder                         0.178999      0.276619      0.386146      0.283977      0.0642911       10     
reco:opslicer:OpSlicer                              0.66622       1.0013        1.32298       1.02732      0.206801        10     
reco:caldata:DataPrepModule                         7.0044        8.98745       9.88355        9.175       0.822402        10     
reco:wclsdatasp:WireCellToolkit                     50.9971       63.7854       67.7166       65.0995       4.56266        10     
reco:gaushit:GausHitFinder                          1.39863       1.98448       2.65612       2.06672      0.392112        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000660359   0.000964365   0.00119516    0.00100665    0.000167435      10     
reco:reco3d:SpacePointSolver                        12.6528       25.1664       40.4317       25.7628       7.09069        10     
reco:hitpdune:DisambigFromSpacePoints               1.89258       3.26142       4.47702       3.34847       0.92376        10     
reco:pandora:StandardPandora                        27.6847       51.5467       71.6415       55.253        13.2739        10     
reco:pandoraWriter:StandardPandora                 0.544169      0.694646      0.880797      0.706884      0.116341        10     
reco:pandoraTrack:LArPandoraTrackCreation           8.8296        13.943        18.1984       14.7619       3.10763        10     
reco:pandoraShower:LArPandoraShowerCreation         6.57451       14.8875       21.1089       16.1161       5.11661        10     
reco:pandoracalo:Calorimetry                        2.95811       6.45504       9.85242       6.02457       2.13779        10     
reco:pandoracalonosce:Calorimetry                   3.54352       5.9518        9.05171       5.97094       1.68379        10     
reco:pandorapid:Chi2ParticleID                    0.00678714    0.00882134     0.0134178     0.0082948     0.0017927       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0729828     0.0920015     0.129066      0.0847338     0.0190424       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00635652    0.00919563     0.0129358    0.00862193    0.00217111       10     
reco:pandoraShowercalo:ShowerCalorimetry            5.89363       14.126        21.4483       16.1326       5.31903        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       4.60861       14.1347       21.1694       16.8716       5.61564        10     
reco:emtrkmichelid:EmTrackMichelId                  220.326       311.808       384.835       313.345       48.6977        10     
reco:crttag:SingleCRTMatchingProducer              0.0260175     0.038215      0.0517026     0.0405232    0.00848492       10     
reco:crtreco:TwoCRTMatchingProducer                0.0206427     0.0320007     0.0394164     0.0352295    0.00666526       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.10209       1.7478        2.20725       1.7296       0.373869        10     
reco:pandora2Track:LArPandoraTrackCreation          21.2559       47.2722       66.7075       53.6089       15.2869        10     
reco:pandora2calo:Calorimetry                       8.63576       20.8808       32.9344       23.2445       7.82946        10     
reco:pandora2calonosce:Calorimetry                  8.19687       19.4093       27.918        21.5571       6.82472        10     
reco:pandora2pid:Chi2ParticleID                    0.0163015     0.024461      0.032801      0.0255257     0.0057264       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0885439     0.121241       0.14961      0.125036      0.0218797       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.016651      0.0231585     0.0306746     0.0236171    0.00529439       10     
reco:pandora2Shower:LArPandoraShowerCreation        7.5372        13.8889       18.0565       15.0537       3.80758        10     
reco:pandora2Showercalo:ShowerCalorimetry           7.59523       16.6997       23.2276       18.6925       5.54419        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      6.44054       14.7423       20.5547       16.6773       4.85604        10     
[art]:TriggerResults:TriggerResultInserter        7.2498e-05    0.000104766   0.000239416   8.6435e-05    4.71188e-05      10     
end_path:out1:RootOutput                            8.9e-06     1.1408e-05    1.7865e-05    1.0938e-05    2.5109e-06       10     
end_path:out1:RootOutput(write)                     3.1576        4.05285       5.03884       4.2165        0.61871        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5225.01 MB
  Peak resident set size usage (VmHWM): 4419.42 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.11 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.35 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.62 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.32 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10426400
-rw-r--r-- 1 pildune22 pildune 2436932459 Jan 11 15:48 detsim_stage2.root
-rw-r--r-- 1 pildune22 pildune 2311376911 Jan 11 15:45 detsim_stage1.root
-rw-r--r-- 1 pildune22 pildune 2132349043 Jan 11 17:44 reco_stage1.root
-rw-r--r-- 1 pildune22 pildune 1896046204 Jan 11 13:38 g4_stage2.root
-rw-r--r-- 1 pildune22 pildune 1822620443 Jan 11 13:18 g4_stage1.root
-rw-r--r-- 1 pildune22 pildune   76691250 Jan 11 17:33 Pandora_Events.pndr
-rw-r--r-- 1 pildune22 pildune     242817 Jan 11 17:44 jobscript.log
-rw-r--r-- 1 pildune22 pildune     217088 Jan 11 17:44 mem.db
-rw-r--r-- 1 pildune22 pildune      62376 Jan 11 17:44 Pandora_Geometry.xml
-rw-r--r-- 1 pildune22 pildune      36864 Jan 11 17:44 time.db
-rw-r--r-- 1 pildune22 pildune        519 Jan 11 15:48 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 pildune22 pildune        519 Jan 11 13:38 g4_protoDUNE_hist.root
-rw-r--r-- 1 pildune22 pildune        519 Jan 11 17:44 protoDUNE_reco1_hist.root
-rw-r--r-- 1 pildune22 pildune         52 Jan 11 13:09 all-input-dids.txt
-rw-r--r-- 1 pildune22 pildune          7 Jan 11 17:44 justin-processed-pfns.txt
-rw-r--r-- 1 pildune22 pildune          0 Jan 11 15:58 ProtoDUNE.log
-rw-r--r-- 1 pildune22 pildune          0 Jan 11 15:48 debugprod.log
justIN time: 2026-02-06 10:21:13 UTC       justIN version: 01.06.00