justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 296574.0@dunegpschedd02.fnal.gov

Jobsub ID296574.0@dunegpschedd02.fnal.gov
Workflow ID12539
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-29 21:26:32
SiteUK_Durham
EntryDUNE_UK_SGridDurham_ce4
Last heartbeat2026-01-30 00:25:19
From worker nodeHostnamen228.dur.scotgrid.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-29 21:27:45
Input filesmonte-carlo-012539-000001
JobscriptExit code0
Real time2h (10562s)
CPU time2h (10322s = 97%)
Max RSS bytes4040245248 (3853 MiB)
Outputting started2026-01-30 00:23:48
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12539/1/001/reco_stage1.root
Finished2026-01-30 00:25:19
Saved logsjustin-logs:296574.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

:   Running tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 89230 traces, 0 summary
Retagger: tagging trace set: gauss with 69237 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 69237 traces tagged "gauss"
FrameSaver: q=2.37479e+06 n=1244048 tag=gauss
wclsFrameSaver: saving 89230 traces tagged "wiener"
FrameSaver: q=2.49908e+06 n=1105735 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 21622 XUs and 23184 XVs -> 235 XUVs
C:0 T:1 41246 XUs and 41951 XVs -> 2478 XUVs
C:0 T:2 29955 XUs and 53639 XVs -> 2285 XUVs
C:0 T:3 13851 XUs and 19418 XVs -> 82 XUVs
C:0 T:4 2438 XUs and 4133 XVs -> 45 XUVs
C:0 T:5 8421 XUs and 10926 XVs -> 2152 XUVs
C:0 T:6 78638 XUs and 71455 XVs -> 2981 XUVs
C:0 T:7 32481 XUs and 27908 XVs -> 182 XUVs
C:0 T:8 5839 XUs and 6370 XVs -> 87 XUVs
C:0 T:9 36832 XUs and 48878 XVs -> 2597 XUVs
C:0 T:10 44492 XUs and 46974 XVs -> 4638 XUVs
C:0 T:11 108 XUs and 103 XVs -> 9 XUVs
17771 XUVs total
8394 collection wire objects
16212 potential space points
Neighbour search...
727296 tests to find 409118 neighbours
Iterating with no regularization...
Begin: 4.30987e+07
0 3.65602e+07
1 3.61335e+07
2 3.60787e+07
3 3.60652e+07
Now with regularization...
Begin: 2.98827e+07
0 2.98713e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 412045421
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 412045421
fLifetime: 30000 [us]
30-Jan-2026 00:23:46 GMT  Closed output file "reco_stage1.root"
30-Jan-2026 00:23:46 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          267.492       389.951       594.912       365.584       107.003        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000886094   0.00280171    0.00443161    0.00292252    0.00112829       10     
reco:ophit:OpHitFinder                             0.244501      0.285536      0.342304      0.285218      0.0288081       10     
reco:opflash:OpFlashFinder                         0.142718      0.207189      0.296363      0.209867      0.0413798       10     
reco:opslicer:OpSlicer                             0.498609      0.684167       0.8828       0.678322      0.118673        10     
reco:caldata:DataPrepModule                         6.12028       6.3127        6.51199       6.30161      0.125496        10     
reco:wclsdatasp:WireCellToolkit                     46.0033       51.6275       55.0442       52.4713       2.30899        10     
reco:gaushit:GausHitFinder                          1.04649       1.34887       2.02494       1.22624      0.292778        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000459951   0.000587442   0.000756819   0.000572747   9.71593e-05      10     
reco:reco3d:SpacePointSolver                        5.44674       13.5328       25.5817       12.1249       6.01079        10     
reco:hitpdune:DisambigFromSpacePoints               1.13624       2.08672       3.54858       1.88493      0.797601        10     
reco:pandora:StandardPandora                        17.5193       33.035        76.8664       28.2805       18.0433        10     
reco:pandoraWriter:StandardPandora                 0.349006      0.466325      0.640899       0.44351      0.090318        10     
reco:pandoraTrack:LArPandoraTrackCreation           2.97219       6.46874       11.2725       6.26827       2.51873        10     
reco:pandoraShower:LArPandoraShowerCreation         2.12722       6.81165       16.0393       5.58062       4.57508        10     
reco:pandoracalo:Calorimetry                        1.50513       2.82811       4.7135        2.79893       1.01011        10     
reco:pandoracalonosce:Calorimetry                   1.17293       2.48797       3.98807       2.49321      0.915984        10     
reco:pandorapid:Chi2ParticleID                    0.00441773    0.00571786    0.00690362    0.00574686    0.000804289      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0420799     0.0639394     0.0884272     0.063761      0.0140509       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00432595    0.00561891    0.00687567    0.00561743    0.000814762      10     
reco:pandoraShowercalo:ShowerCalorimetry            1.92041       5.83321       12.686        4.6765        3.48556        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.9828        5.50869       12.1235       4.17945       3.54027        10     
reco:emtrkmichelid:EmTrackMichelId                  140.498       191.095       260.001       184.001       38.6545        10     
reco:crttag:SingleCRTMatchingProducer              0.0145871     0.022407      0.0303728     0.0220662    0.00418787       10     
reco:crtreco:TwoCRTMatchingProducer               0.00970919     0.0160372     0.0280395     0.0152747    0.00489562       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.728606       1.23728       1.63894       1.30916      0.275343        10     
reco:pandora2Track:LArPandoraTrackCreation          8.57873       20.5964       38.3608       19.1973       10.1989        10     
reco:pandora2calo:Calorimetry                       3.33883       8.09713       15.4372       6.98808       4.07669        10     
reco:pandora2calonosce:Calorimetry                  2.89121       7.55065       15.3528       6.65886       4.10339        10     
reco:pandora2pid:Chi2ParticleID                   0.00968227     0.0140337     0.0207018     0.0128126     0.0035655       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0544831     0.0805571     0.116885      0.0757052     0.0199214       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00976885     0.0139934     0.0208621     0.0128805     0.0034812       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.67142       6.02506       11.3434       5.23676       2.88138        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.69626       6.76844        12.62        6.49088       3.27139        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.61076       5.95317       10.945        5.91837       2.69852        10     
[art]:TriggerResults:TriggerResultInserter        3.9582e-05    5.62384e-05   0.000106962   5.26395e-05   1.78101e-05      10     
end_path:out1:RootOutput                           7.466e-06    8.4073e-06    1.1951e-05     8.075e-06    1.21387e-06      10     
end_path:out1:RootOutput(write)                     1.88583       2.75577       3.89345       2.7403       0.627367        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4849.63 MB
  Peak resident set size usage (VmHWM): 4040.25 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.22 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.98 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.22 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9532464
-rw-r--r--. 1 dune004 dune 2256951678 Jan 29 23:16 detsim_stage2.root
-rw-r--r--. 1 dune004 dune 2145285319 Jan 29 23:14 detsim_stage1.root
-rw-r--r--. 1 dune004 dune 1921958688 Jan 30 00:23 reco_stage1.root
-rw-r--r--. 1 dune004 dune 1709720672 Jan 29 21:48 g4_stage2.root
-rw-r--r--. 1 dune004 dune 1657214114 Jan 29 21:34 g4_stage1.root
-rw-r--r--. 1 dune004 dune   69477450 Jan 30 00:20 Pandora_Events.pndr
-rw-r--r--. 1 dune004 dune     242658 Jan 30 00:23 jobscript.log
-rw-r--r--. 1 dune004 dune     217088 Jan 30 00:23 mem.db
-rw-r--r--. 1 dune004 dune      62376 Jan 30 00:23 Pandora_Geometry.xml
-rw-r--r--. 1 dune004 dune      36864 Jan 30 00:23 time.db
-rw-r--r--. 1 dune004 dune        519 Jan 29 23:16 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune004 dune        519 Jan 29 21:48 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune004 dune        519 Jan 30 00:23 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune004 dune         52 Jan 29 21:27 all-input-dids.txt
-rw-r--r--. 1 dune004 dune          7 Jan 30 00:23 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune          0 Jan 29 23:22 ProtoDUNE.log
-rw-r--r--. 1 dune004 dune          0 Jan 29 23:16 debugprod.log
justIN time: 2026-02-04 04:27:10 UTC       justIN version: 01.06.00