justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 260266.1@dunegpschedd02.fnal.gov

Jobsub ID260266.1@dunegpschedd02.fnal.gov
Workflow ID11141
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-10 13:59:26
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-12-10 23:59:02
From worker nodeHostnamewn1207091.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-10 18:29:08
Input filesmonte-carlo-011141-000001
JobscriptExit code0
Real time5h (19690s)
CPU time5h (19029s = 96%)
Max RSS bytes4334432256 (4133 MiB)
Outputting started2025-12-10 23:57:19
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11141/1/001/reco_stage1.root
Finished2025-12-10 23:59:02
Saved logsjustin-logs:260266.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 156866 traces, 0 summary
Retagger: tagging trace set: gauss with 118343 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 118343 traces tagged "gauss"
FrameSaver: q=4.25834e+06 n=2089107 tag=gauss
wclsFrameSaver: saving 156866 traces tagged "wiener"
FrameSaver: q=4.50656e+06 n=1846747 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 43407 XUs and 40290 XVs -> 351 XUVs
C:0 T:1 108244 XUs and 92140 XVs -> 12125 XUVs
C:0 T:2 135818 XUs and 155096 XVs -> 7902 XUVs
C:0 T:3 60293 XUs and 64751 XVs -> 507 XUVs
C:0 T:4 18530 XUs and 18626 XVs -> 280 XUVs
C:0 T:5 94573 XUs and 85105 XVs -> 6493 XUVs
C:0 T:6 64502 XUs and 65442 XVs -> 2894 XUVs
C:0 T:7 29372 XUs and 28410 XVs -> 281 XUVs
C:0 T:8 9815 XUs and 10153 XVs -> 109 XUVs
C:0 T:9 58504 XUs and 76459 XVs -> 4079 XUVs
C:0 T:10 53420 XUs and 20391 XVs -> 6947 XUVs
C:0 T:11 65 XUs and 80 XVs -> 29 XUVs
41997 XUVs total
13985 collection wire objects
38330 potential space points
Neighbour search...
2718426 tests to find 1383358 neighbours
Iterating with no regularization...
Begin: 1.048e+08
0 9.03515e+07
1 8.91841e+07
2 8.90131e+07
3 8.89613e+07
Now with regularization...
Begin: 7.55775e+07
0 7.55472e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411032529
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411032529
fLifetime: 30000 [us]
10-Dec-2025 23:57:16 GMT  Closed output file "reco_stage1.root"
10-Dec-2025 23:57:16 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          552.797       674.595       931.595       649.743       103.577        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.0015683    0.00451104    0.00773069    0.00464733    0.00194981       10     
reco:ophit:OpHitFinder                             0.334595      0.440824      0.592636      0.457757      0.079075        10     
reco:opflash:OpFlashFinder                         0.201177      0.269752      0.368841      0.261942      0.0574835       10     
reco:opslicer:OpSlicer                             0.854863       1.11864       1.37245       1.15067      0.156741        10     
reco:caldata:DataPrepModule                         6.44726       8.01024       9.96864       7.83323       1.04671        10     
reco:wclsdatasp:WireCellToolkit                     48.2579       56.1131       67.9785       56.3247       5.34135        10     
reco:gaushit:GausHitFinder                          1.20594       2.02117       2.89298       1.96736      0.523838        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000608409   0.000986314   0.00165209    0.000956916   0.000311134      10     
reco:reco3d:SpacePointSolver                        14.9818       24.2813       44.3382       23.8557       7.86728        10     
reco:hitpdune:DisambigFromSpacePoints               1.82937       2.72626       4.71887       2.48703      0.794483        10     
reco:pandora:StandardPandora                        32.7275       57.3765       102.919       49.6544       22.0442        10     
reco:pandoraWriter:StandardPandora                 0.523842      0.655718       1.04012       0.59843      0.149086        10     
reco:pandoraTrack:LArPandoraTrackCreation           6.03571       8.31471       12.0601       8.01446       1.71658        10     
reco:pandoraShower:LArPandoraShowerCreation         6.31047       10.016        17.0063       9.13405       2.98947        10     
reco:pandoracalo:Calorimetry                        3.2846        4.9531        7.78696       4.09779       1.60141        10     
reco:pandoracalonosce:Calorimetry                   3.27618       4.70464       6.98126       4.77871       1.15823        10     
reco:pandorapid:Chi2ParticleID                    0.00696258     0.010182      0.0151362     0.0103179    0.00222093       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0605742      0.09824       0.16563      0.0969783     0.0287196       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00630991    0.00962664     0.0122139     0.0103014    0.00169152       10     
reco:pandoraShowercalo:ShowerCalorimetry            7.44181       11.3186       17.9453       10.5084       3.18856        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       6.65468       10.3712       17.0434       9.3048        3.47865        10     
reco:emtrkmichelid:EmTrackMichelId                  313.986       374.922       489.108       356.739       48.4079        10     
reco:crttag:SingleCRTMatchingProducer              0.0154318     0.0268058     0.035893      0.0274917     0.0055733       10     
reco:crtreco:TwoCRTMatchingProducer                0.0119055     0.0207217     0.0321016     0.0195093    0.00564448       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.12408       1.49893       2.07954       1.48464      0.258363        10     
reco:pandora2Track:LArPandoraTrackCreation          21.7617       28.6323       42.5706       27.7154       6.04291        10     
reco:pandora2calo:Calorimetry                       10.3756       14.5044       21.0202       13.1707       3.45073        10     
reco:pandora2calonosce:Calorimetry                  9.68152       13.7361       19.117        13.0132       3.24482        10     
reco:pandora2pid:Chi2ParticleID                    0.0169743     0.0232146     0.0297969     0.0229313    0.00401767       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.079013      0.119158       0.16153      0.132227      0.0284739       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0178659     0.0256362     0.0382151     0.0262516    0.00602368       10     
reco:pandora2Shower:LArPandoraShowerCreation        7.22368       10.0089       15.2135       9.58289       2.38824        10     
reco:pandora2Showercalo:ShowerCalorimetry           8.18701       12.5078       17.552        11.9622       3.20129        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      7.21043       10.7126       15.2448       10.2056       2.74394        10     
[art]:TriggerResults:TriggerResultInserter        3.4305e-05    5.58455e-05   0.000187194    3.728e-05    4.46841e-05      10     
end_path:out1:RootOutput                           6.443e-06    7.9119e-06    1.5529e-05    7.0125e-06    2.61606e-06      10     
end_path:out1:RootOutput(write)                     3.83267       4.84642       7.14274       4.26369       1.03633        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5183.24 MB
  Peak resident set size usage (VmHWM): 4334.43 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.34 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.16 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.21 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10452700
-rw-r--r-- 1 dune01 dune01 2437215105 Dec 10 22:01 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2312095369 Dec 10 21:58 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2150959067 Dec 10 23:57 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1894573462 Dec 10 19:00 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1826046203 Dec 10 18:39 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   82042310 Dec 10 23:47 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     243098 Dec 10 23:57 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Dec 10 23:57 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Dec 10 23:57 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Dec 10 23:57 time.db
-rw-r--r-- 1 dune01 dune01        519 Dec 10 22:01 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec 10 19:00 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Dec 10 23:57 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Dec 10 18:29 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Dec 10 23:57 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Dec 10 22:15 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Dec 10 22:01 debugprod.log
justIN time: 2026-02-07 13:30:00 UTC       justIN version: 01.06.00