justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 302154.1@dunegpschedd01.fnal.gov

Jobsub ID302154.1@dunegpschedd01.fnal.gov
Workflow ID12532
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-29 16:40:09
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-01-29 20:48:12
From worker nodeHostnamewn2208241.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-29 16:40:45
Input filesmonte-carlo-012532-000001
JobscriptExit code0
Real time4h (14754s)
CPU time3h (14238s = 96%)
Max RSS bytes4087906304 (3898 MiB)
Outputting started2026-01-29 20:46:41
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12532/1/001/reco_stage1.root
Finished2026-01-29 20:48:12
Saved logsjustin-logs:302154.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

KeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 137284 traces, 0 summary
Retagger: tagging trace set: gauss with 106425 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 106425 traces tagged "gauss"
FrameSaver: q=3.57626e+06 n=1883451 tag=gauss
wclsFrameSaver: saving 137284 traces tagged "wiener"
FrameSaver: q=3.78223e+06 n=1674990 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 34136 XUs and 33349 XVs -> 588 XUVs
C:0 T:1 69892 XUs and 57982 XVs -> 4745 XUVs
C:0 T:2 103315 XUs and 144824 XVs -> 8551 XUVs
C:0 T:3 47483 XUs and 56718 XVs -> 720 XUVs
C:0 T:4 12491 XUs and 12689 XVs -> 118 XUVs
C:0 T:5 47415 XUs and 39916 XVs -> 3868 XUVs
C:0 T:6 124783 XUs and 119230 XVs -> 6223 XUVs
C:0 T:7 50471 XUs and 46785 XVs -> 425 XUVs
C:0 T:8 7441 XUs and 10788 XVs -> 120 XUVs
C:0 T:9 57262 XUs and 160281 XVs -> 7853 XUVs
C:0 T:10 39259 XUs and 15304 XVs -> 3368 XUVs
C:0 T:11 401 XUs and 217 XVs -> 33 XUVs
36612 XUVs total
12331 collection wire objects
33081 potential space points
Neighbour search...
2668437 tests to find 1335904 neighbours
Iterating with no regularization...
Begin: 6.73833e+07
0 5.16454e+07
1 5.07252e+07
2 5.05733e+07
3 5.05241e+07
Now with regularization...
Begin: 4.06295e+07
0 4.05938e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 412041426
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 412041426
fLifetime: 30000 [us]
29-Jan-2026 20:46:36 GMT  Closed output file "reco_stage1.root"
29-Jan-2026 20:46:36 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          311.322       398.043       483.388       413.428       58.5499        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00146219    0.00494497    0.00984878    0.00488172     0.0023044       10     
reco:ophit:OpHitFinder                             0.319347      0.363138      0.449653      0.356409      0.0351283       10     
reco:opflash:OpFlashFinder                         0.142463      0.174523      0.217519      0.170176      0.025128        10     
reco:opslicer:OpSlicer                             0.639451      0.845466       1.06143      0.839985      0.104979        10     
reco:caldata:DataPrepModule                         5.68838       7.40318       8.8325        7.78344       0.98614        10     
reco:wclsdatasp:WireCellToolkit                     37.197        46.5653       52.6982       47.3148       3.86617        10     
reco:gaushit:GausHitFinder                          1.20495       1.49181       1.77326       1.45487      0.177777        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000385044   0.000669664   0.000908669   0.000649903   0.000145248      10     
reco:reco3d:SpacePointSolver                        13.0563       18.3708       23.8349       18.9066       3.84287        10     
reco:hitpdune:DisambigFromSpacePoints               1.24843       1.89458       2.50624       1.86941      0.434876        10     
reco:pandora:StandardPandora                        20.9713       35.7758       51.5886       35.9865       9.74233        10     
reco:pandoraWriter:StandardPandora                 0.499237      0.591784      0.722103      0.565505      0.0843087       10     
reco:pandoraTrack:LArPandoraTrackCreation           2.58888       4.00686       5.57083       4.10057      0.877174        10     
reco:pandoraShower:LArPandoraShowerCreation         2.46797       4.4545        6.39865       4.83348       1.29263        10     
reco:pandoracalo:Calorimetry                        1.74173       2.51565       3.16071       2.67109      0.532492        10     
reco:pandoracalonosce:Calorimetry                   1.4373        2.25931       2.92835       2.35966      0.503861        10     
reco:pandorapid:Chi2ParticleID                    0.00589557    0.00776388    0.00922586    0.00829682    0.000975993      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0361059     0.0730129     0.0862078     0.079802      0.0142341       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00361735    0.00699153    0.00836057    0.00742512    0.00143157       10     
reco:pandoraShowercalo:ShowerCalorimetry            2.0493        4.8894        6.89775       5.53092       1.53729        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.0298        4.76236       6.74327       5.18553       1.45197        10     
reco:emtrkmichelid:EmTrackMichelId                  167.246       215.66        254.824       222.612       28.1606        10     
reco:crttag:SingleCRTMatchingProducer              0.0100256     0.016656      0.0234817     0.0169987    0.00431655       10     
reco:crtreco:TwoCRTMatchingProducer               0.00537407    0.00903872     0.0115011     0.0100397    0.00228855       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.704892      0.957615       1.20626      0.988349      0.140869        10     
reco:pandora2Track:LArPandoraTrackCreation          6.29558       12.5822       17.0053       13.2216       3.88748        10     
reco:pandora2calo:Calorimetry                       3.1582        6.58799       9.17597       7.24262        2.085         10     
reco:pandora2calonosce:Calorimetry                  3.35875       6.24945       8.58803       7.1896        1.94711        10     
reco:pandora2pid:Chi2ParticleID                   0.00821927     0.0174398     0.0219736     0.0191059    0.00407113       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0440911     0.0883049     0.111426      0.0937426     0.0188428       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00786937     0.0170218     0.021483      0.0180758    0.00404819       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.5171        4.66714       6.59631       5.12469       1.34662        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.14866       5.68596       8.13878       6.1863        1.77071        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.79626       5.26954       7.27873       6.01502       1.60212        10     
[art]:TriggerResults:TriggerResultInserter        7.5792e-05    0.000122672   0.000327606   9.3696e-05    7.13978e-05      10     
end_path:out1:RootOutput                          1.2143e-05    1.80568e-05   4.4524e-05    1.3906e-05    9.32707e-06      10     
end_path:out1:RootOutput(write)                     2.9435        3.63954       4.36073       3.65095      0.447357        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4944.01 MB
  Peak resident set size usage (VmHWM): 4087.91 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.14 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.35 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.36 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.19 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9688896
-rw-r--r-- 1 dune01 dune01 2297912334 Jan 29 19:37 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2183735128 Jan 29 19:34 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 1967511382 Jan 29 20:46 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1738455532 Jan 29 17:02 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1662706461 Jan 29 16:48 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   70479330 Jan 29 20:40 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242718 Jan 29 20:46 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Jan 29 20:46 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Jan 29 20:46 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Jan 29 20:46 time.db
-rw-r--r-- 1 dune01 dune01        519 Jan 29 19:37 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 29 17:02 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Jan 29 20:46 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Jan 29 16:40 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Jan 29 20:46 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Jan 29 19:43 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Jan 29 19:37 debugprod.log
justIN time: 2026-02-04 15:17:07 UTC       justIN version: 01.06.00