justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 302386.0@dunegpschedd01.fnal.gov

Jobsub ID302386.0@dunegpschedd01.fnal.gov
Workflow ID12576
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-30 13:59:13
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat2026-01-30 17:32:54
From worker nodeHostnameheplnc166.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-30 14:00:49
Input filesmonte-carlo-012576-000001
JobscriptExit code0
Real time3h (12478s)
CPU time3h (11847s = 94%)
Max RSS bytes4052123648 (3864 MiB)
Outputting started2026-01-30 17:28:48
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12576/1/001/reco_stage1.root
Finished2026-01-30 17:32:54
Saved logsjustin-logs:302386.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ng tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 158958 traces, 0 summary
Retagger: tagging trace set: gauss with 121868 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 121868 traces tagged "gauss"
FrameSaver: q=4.28626e+06 n=2190139 tag=gauss
wclsFrameSaver: saving 158958 traces tagged "wiener"
FrameSaver: q=4.52844e+06 n=1941473 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 58346 XUs and 58474 XVs -> 1478 XUVs
C:0 T:1 145499 XUs and 134836 XVs -> 17446 XUVs
C:0 T:2 141795 XUs and 139135 XVs -> 4832 XUVs
C:0 T:3 69086 XUs and 63144 XVs -> 601 XUVs
C:0 T:4 12515 XUs and 14475 XVs -> 88 XUVs
C:0 T:5 57533 XUs and 49912 XVs -> 7132 XUVs
C:0 T:6 151930 XUs and 178501 XVs -> 7201 XUVs
C:0 T:7 56162 XUs and 64910 XVs -> 759 XUVs
C:0 T:8 7853 XUs and 6559 XVs -> 48 XUVs
C:0 T:9 49422 XUs and 73247 XVs -> 6007 XUVs
C:0 T:10 21957 XUs and 9472 XVs -> 1883 XUVs
C:0 T:11 148 XUs and 133 XVs -> 16 XUVs
47491 XUVs total
13716 collection wire objects
42298 potential space points
Neighbour search...
4146000 tests to find 2043776 neighbours
Iterating with no regularization...
Begin: 7.5639e+07
0 6.23961e+07
1 6.13399e+07
2 6.11956e+07
3 6.11523e+07
Now with regularization...
Begin: 4.9695e+07
0 4.96644e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 412059345
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 412059345
fLifetime: 30000 [us]
30-Jan-2026 17:28:46 GMT  Closed output file "reco_stage1.root"
30-Jan-2026 17:28:46 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          245.081       367.627       603.223       332.687       103.045        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00142094    0.00360804    0.00626152    0.00319146    0.00144238       10     
reco:ophit:OpHitFinder                              0.20997      0.322354      0.454667      0.333426      0.0667266       10     
reco:opflash:OpFlashFinder                         0.114449      0.172978      0.282106      0.175231      0.042425        10     
reco:opslicer:OpSlicer                             0.404829      0.739922       1.18142      0.719172      0.228314        10     
reco:caldata:DataPrepModule                         4.79072       6.70497       7.93947       6.87433       1.11357        10     
reco:wclsdatasp:WireCellToolkit                     31.0553       37.5011       44.5439       36.7269       4.8974         10     
reco:gaushit:GausHitFinder                         0.732879       1.32903       2.27268       1.08401      0.497734        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000348332   0.000581039   0.00088138    0.000544991   0.00015117       10     
reco:reco3d:SpacePointSolver                        10.0599       16.7037       30.0768       14.8216       6.26498        10     
reco:hitpdune:DisambigFromSpacePoints              0.855714       1.53209       2.59132       1.35586      0.560107        10     
reco:pandora:StandardPandora                        14.5568       36.3785       104.864       30.6249       24.4443        10     
reco:pandoraWriter:StandardPandora                 0.315664      0.531604      0.725972      0.532264      0.136138        10     
reco:pandoraTrack:LArPandoraTrackCreation           2.4006        3.73831       6.78946       3.40405       1.27999        10     
reco:pandoraShower:LArPandoraShowerCreation         1.72543       3.72127       9.08838       3.1895        2.07522        10     
reco:pandoracalo:Calorimetry                       0.957019       2.14935       4.07284       2.08031      0.804179        10     
reco:pandoracalonosce:Calorimetry                  0.883159       1.8741        3.61993       1.81703      0.691406        10     
reco:pandorapid:Chi2ParticleID                    0.00361519    0.00622057    0.00941912    0.00559385    0.00218873       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.029066      0.0625457     0.0950517     0.0635171     0.021358        10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00383065    0.00582766    0.00816389    0.00538213    0.00171707       10     
reco:pandoraShowercalo:ShowerCalorimetry            1.58692       3.95031       9.89368       3.65692       2.16901        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.34702       3.72809       8.96704        3.611        1.97002        10     
reco:emtrkmichelid:EmTrackMichelId                  145.304       203.142       285.887       196.11        44.4394        10     
reco:crttag:SingleCRTMatchingProducer             0.00860922     0.0167698     0.0218273     0.018289     0.00451156       10     
reco:crtreco:TwoCRTMatchingProducer                0.0055784     0.0101206     0.0153227     0.0102216    0.00262109       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.681226       1.00766       1.3428        1.00214      0.198086        10     
reco:pandora2Track:LArPandoraTrackCreation          6.0654        12.666        24.9932       10.8666       5.81749        10     
reco:pandora2calo:Calorimetry                       3.17049       6.22039       11.3586       5.47064       2.76563        10     
reco:pandora2calonosce:Calorimetry                  2.90379       5.97222       11.7727       4.53592       2.95393        10     
reco:pandora2pid:Chi2ParticleID                   0.00866676     0.0149045     0.023754      0.0131319    0.00563738       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0464003     0.079311      0.128604      0.0705992     0.029643        10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0089337     0.0146524     0.0240135     0.0131079    0.00550017       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.36117       4.19313       8.45911       2.87162       2.08064        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.64952       5.27068       10.2483       3.7414        2.64292        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.42727       4.50854       7.58077       3.45262       1.99661        10     
[art]:TriggerResults:TriggerResultInserter        5.1015e-05    7.36218e-05   0.000120285   7.4059e-05    1.97643e-05      10     
end_path:out1:RootOutput                           8.977e-06    1.13112e-05   1.4988e-05     1.079e-05    2.0463e-06       10     
end_path:out1:RootOutput(write)                     2.18275       3.22943       4.87005       3.12392      0.907967        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4872.76 MB
  Peak resident set size usage (VmHWM): 4052.12 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.30 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.32 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.18 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9296820
-rw-r--r-- 1 pltdune04 pltdune 2216524034 Jan 30 16:25 detsim_stage2.root
-rw-r--r-- 1 pltdune04 pltdune 2105078441 Jan 30 16:23 detsim_stage1.root
-rw-r--r-- 1 pltdune04 pltdune 1880195912 Jan 30 17:28 reco_stage1.root
-rw-r--r-- 1 pltdune04 pltdune 1656944820 Jan 30 14:22 g4_stage2.root
-rw-r--r-- 1 pltdune04 pltdune 1590784732 Jan 30 14:10 g4_stage1.root
-rw-r--r-- 1 pltdune04 pltdune   69601750 Jan 30 17:21 Pandora_Events.pndr
-rw-r--r-- 1 pltdune04 pltdune     242697 Jan 30 17:28 jobscript.log
-rw-r--r-- 1 pltdune04 pltdune     217088 Jan 30 17:28 mem.db
-rw-r--r-- 1 pltdune04 pltdune      62376 Jan 30 17:28 Pandora_Geometry.xml
-rw-r--r-- 1 pltdune04 pltdune      36864 Jan 30 17:28 time.db
-rw-r--r-- 1 pltdune04 pltdune        519 Jan 30 16:25 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 pltdune04 pltdune        519 Jan 30 14:22 g4_protoDUNE_hist.root
-rw-r--r-- 1 pltdune04 pltdune        519 Jan 30 17:28 protoDUNE_reco1_hist.root
-rw-r--r-- 1 pltdune04 pltdune         52 Jan 30 14:00 all-input-dids.txt
-rw-r--r-- 1 pltdune04 pltdune          7 Jan 30 17:28 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune04 pltdune          0 Jan 30 16:31 ProtoDUNE.log
-rw-r--r-- 1 pltdune04 pltdune          0 Jan 30 16:25 debugprod.log
justIN time: 2026-02-04 10:45:38 UTC       justIN version: 01.06.00