justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 296575.0@dunegpschedd02.fnal.gov

Jobsub ID296575.0@dunegpschedd02.fnal.gov
Workflow ID12540
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-29 21:26:32
SiteUK_Durham
EntryDUNE_UK_SGridDurham_ce3
Last heartbeat2026-01-30 00:19:13
From worker nodeHostnamen174.dur.scotgrid.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-29 21:29:51
Input filesmonte-carlo-012540-000001
JobscriptExit code0
Real time2h (10065s)
CPU time2h (9871s = 98%)
Max RSS bytes3982106624 (3797 MiB)
Outputting started2026-01-30 00:17:37
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12540/1/001/reco_stage1.root
Finished2026-01-30 00:19:13
Saved logsjustin-logs:296575.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

g tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 132285 traces, 0 summary
Retagger: tagging trace set: gauss with 96972 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 96972 traces tagged "gauss"
FrameSaver: q=3.24371e+06 n=1661262 tag=gauss
wclsFrameSaver: saving 132285 traces tagged "wiener"
FrameSaver: q=3.43197e+06 n=1439315 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 16951 XUs and 22407 XVs -> 151 XUVs
C:0 T:1 40237 XUs and 41787 XVs -> 2621 XUVs
C:0 T:2 43585 XUs and 76984 XVs -> 3529 XUVs
C:0 T:3 20289 XUs and 30389 XVs -> 100 XUVs
C:0 T:4 14537 XUs and 19777 XVs -> 142 XUVs
C:0 T:5 70074 XUs and 70288 XVs -> 5523 XUVs
C:0 T:6 118765 XUs and 133972 XVs -> 8266 XUVs
C:0 T:7 41317 XUs and 46748 XVs -> 440 XUVs
C:0 T:8 7359 XUs and 9822 XVs -> 82 XUVs
C:0 T:9 51606 XUs and 102326 XVs -> 6757 XUVs
C:0 T:10 39986 XUs and 27490 XVs -> 6498 XUVs
C:0 T:11 53 XUs and 22 XVs -> 4 XUVs
34113 XUVs total
12438 collection wire objects
32049 potential space points
Neighbour search...
3076057 tests to find 1468788 neighbours
Iterating with no regularization...
Begin: 6.23096e+07
0 5.16666e+07
1 5.09942e+07
2 5.08883e+07
3 5.08565e+07
Now with regularization...
Begin: 4.21811e+07
0 4.2147e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 412045434
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 412045434
fLifetime: 30000 [us]
30-Jan-2026 00:17:35 GMT  Closed output file "reco_stage1.root"
30-Jan-2026 00:17:35 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          288.117       373.201       584.725       355.109       81.0483        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000925124   0.00395568    0.00830891    0.00352299    0.00222309       10     
reco:ophit:OpHitFinder                             0.258739      0.296543      0.344196       0.29584      0.0258789       10     
reco:opflash:OpFlashFinder                         0.165426      0.207772      0.284646      0.197089      0.0363151       10     
reco:opslicer:OpSlicer                             0.602594      0.690062      0.928325      0.646579      0.0975753       10     
reco:caldata:DataPrepModule                         6.13334       6.44353       6.86353       6.4373       0.189731        10     
reco:wclsdatasp:WireCellToolkit                     46.1706       51.0247       54.5274       51.4976       2.46493        10     
reco:gaushit:GausHitFinder                          1.0998        1.3606        2.03675       1.31097      0.263183        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000485233   0.000611085   0.000840436   0.000551337   0.000126653      10     
reco:reco3d:SpacePointSolver                        9.77728       15.1338       23.9853       13.8775       4.62502        10     
reco:hitpdune:DisambigFromSpacePoints               1.36655       1.89233       3.39586       1.76152      0.555536        10     
reco:pandora:StandardPandora                        20.7865       29.8653       63.8017       28.6232       12.0886        10     
reco:pandoraWriter:StandardPandora                 0.375002      0.461358      0.641273      0.439575      0.0746594       10     
reco:pandoraTrack:LArPandoraTrackCreation           4.10945       6.09898       12.2116       5.69266       2.2588         10     
reco:pandoraShower:LArPandoraShowerCreation         3.08843       5.88695       14.2763       5.25433       3.1086         10     
reco:pandoracalo:Calorimetry                        1.89085       2.73476       5.49237       2.55366       1.00355        10     
reco:pandoracalonosce:Calorimetry                   1.69912       2.50183       4.96591       2.34413       0.91118        10     
reco:pandorapid:Chi2ParticleID                    0.00518519    0.00582673    0.00728409    0.00544248    0.000704728      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0548106     0.063622      0.0777441     0.0617133     0.0077573       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00502616    0.00555962    0.00710304     0.0053646    0.000575591      10     
reco:pandoraShowercalo:ShowerCalorimetry            2.82793       5.42283       12.7289       4.56236       2.84176        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.46056       5.1685        12.0381       4.4968        2.75314        10     
reco:emtrkmichelid:EmTrackMichelId                  150.862       183.605       258.168       175.155       30.3248        10     
reco:crttag:SingleCRTMatchingProducer              0.0162735     0.0217688     0.0314843     0.019401     0.00496179       10     
reco:crtreco:TwoCRTMatchingProducer                0.0109507     0.0144435     0.022113      0.013597     0.00336079       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.937675       1.20567       1.40661       1.18576      0.148833        10     
reco:pandora2Track:LArPandoraTrackCreation          10.9786       18.8194       41.2508       16.5884       8.50788        10     
reco:pandora2calo:Calorimetry                       4.34062       7.49951       16.7229       6.50217       3.57549        10     
reco:pandora2calonosce:Calorimetry                  3.85818       6.83009       14.7753       5.98655       3.14006        10     
reco:pandora2pid:Chi2ParticleID                    0.0106268     0.0136042     0.0211069     0.0130315    0.00273275       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0651952     0.0785943     0.109623      0.0743326     0.0133902       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.010648      0.0134315     0.0204112     0.0128547    0.00260067       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.31559       5.54389       10.6343       4.9743        2.14877        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.52758       6.04847       11.4776       5.41212       2.41884        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.16103       5.43018       10.5321       4.7969        2.21429        10     
[art]:TriggerResults:TriggerResultInserter        4.5373e-05    6.94465e-05   0.000132679   6.21795e-05   2.47658e-05      10     
end_path:out1:RootOutput                           7.663e-06    1.02066e-05   1.4171e-05    9.3275e-06    2.28279e-06      10     
end_path:out1:RootOutput(write)                     2.13411       2.68483       3.96278       2.55109      0.529959        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4786.38 MB
  Peak resident set size usage (VmHWM): 3982.11 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.12 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.24 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.94 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.22 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9208260
-rw-r--r--. 1 dune004 dune 2199558714 Jan 29 23:13 detsim_stage2.root
-rw-r--r--. 1 dune004 dune 2087099742 Jan 29 23:11 detsim_stage1.root
-rw-r--r--. 1 dune004 dune 1850052654 Jan 30 00:17 reco_stage1.root
-rw-r--r--. 1 dune004 dune 1647916693 Jan 29 21:48 g4_stage2.root
-rw-r--r--. 1 dune004 dune 1577926899 Jan 29 21:36 g4_stage1.root
-rw-r--r--. 1 dune004 dune   66071520 Jan 30 00:12 Pandora_Events.pndr
-rw-r--r--. 1 dune004 dune     242912 Jan 30 00:17 jobscript.log
-rw-r--r--. 1 dune004 dune     217088 Jan 30 00:17 mem.db
-rw-r--r--. 1 dune004 dune      62376 Jan 30 00:17 Pandora_Geometry.xml
-rw-r--r--. 1 dune004 dune      36864 Jan 30 00:17 time.db
-rw-r--r--. 1 dune004 dune        519 Jan 29 23:13 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune004 dune        519 Jan 29 21:48 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune004 dune        519 Jan 30 00:17 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune004 dune         52 Jan 29 21:29 all-input-dids.txt
-rw-r--r--. 1 dune004 dune          7 Jan 30 00:17 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune          0 Jan 29 23:19 ProtoDUNE.log
-rw-r--r--. 1 dune004 dune          0 Jan 29 23:13 debugprod.log
justIN time: 2026-02-04 04:31:48 UTC       justIN version: 01.06.00