justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 14106.1@dunegpschedd02.fnal.gov

Jobsub ID14106.1@dunegpschedd02.fnal.gov
Workflow ID263
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-08-03 07:30:14
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-08-03 15:24:46
From worker nodeHostnamewn1206080.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-03 07:31:28
Input filesmonte-carlo-000263-000001
JobscriptExit code0
Real time7h (28216s)
CPU time7h (26344s = 93%)
Max RSS bytes4361490432 (4159 MiB)
Outputting started2025-08-03 15:21:45
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/00263/1/001/reco_stage1.root
Finished2025-08-03 15:24:46
Saved logsjustin-logs:14106.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ing tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 120084 traces, 0 summary
Retagger: tagging trace set: gauss with 94149 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 94149 traces tagged "gauss"
FrameSaver: q=3.26485e+06 n=1675361 tag=gauss
wclsFrameSaver: saving 120084 traces tagged "wiener"
FrameSaver: q=3.44666e+06 n=1499167 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 24354 XUs and 22091 XVs -> 135 XUVs
C:0 T:1 39177 XUs and 35239 XVs -> 2743 XUVs
C:0 T:2 124457 XUs and 161535 XVs -> 7041 XUVs
C:0 T:3 59156 XUs and 59857 XVs -> 437 XUVs
C:0 T:4 10127 XUs and 8926 XVs -> 93 XUVs
C:0 T:5 30266 XUs and 23785 XVs -> 4715 XUVs
C:0 T:6 36336 XUs and 51009 XVs -> 1355 XUVs
C:0 T:7 17260 XUs and 23229 XVs -> 477 XUVs
C:0 T:8 8132 XUs and 9887 XVs -> 81 XUVs
C:0 T:9 46201 XUs and 67048 XVs -> 3019 XUVs
C:0 T:10 38026 XUs and 31266 XVs -> 3851 XUVs
C:0 T:11 70 XUs and 43 XVs -> 4 XUVs
23951 XUVs total
10017 collection wire objects
22450 potential space points
Neighbour search...
1082500 tests to find 603024 neighbours
Iterating with no regularization...
Begin: 7.67885e+07
0 6.57936e+07
1 6.52271e+07
2 6.51516e+07
3 6.51338e+07
Now with regularization...
Begin: 5.55915e+07
0 5.55708e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 408432827
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 408432827
fLifetime: 30000 [us]
03-Aug-2025 16:21:41 BST  Closed output file "reco_stage1.root"
03-Aug-2025 16:21:41 BST  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          822.215       1071.13       1428.59       991.348       183.943        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.0032419     0.0121755     0.0284639    0.00890747    0.00751193       10     
reco:ophit:OpHitFinder                             0.427532      0.489048      0.629485      0.472478      0.0564998       10     
reco:opflash:OpFlashFinder                         0.235753      0.320036       0.4912       0.283031      0.0867818       10     
reco:opslicer:OpSlicer                              1.11746       1.39861       1.85699       1.32663      0.229762        10     
reco:caldata:DataPrepModule                         9.90237       10.3868       11.0507       10.2783      0.321189        10     
reco:wclsdatasp:WireCellToolkit                     79.2333       89.9082       106.85        90.9691       8.23884        10     
reco:gaushit:GausHitFinder                          2.28754       2.98335       3.61827       3.06446      0.437119        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00141001     0.0024598    0.00366948    0.00234395    0.000701153      10     
reco:reco3d:SpacePointSolver                        18.4114       31.7634       68.3077       28.0334       13.5015        10     
reco:hitpdune:DisambigFromSpacePoints               2.22124       3.69514       5.33658       3.49332      0.976467        10     
reco:pandora:StandardPandora                        59.823        106.35        179.467       94.7961       38.1345        10     
reco:pandoraWriter:StandardPandora                 0.983274       1.2511        1.72999       1.28491      0.221285        10     
reco:pandoraTrack:LArPandoraTrackCreation           14.5209       19.9727       25.7495       19.1187       3.86315        10     
reco:pandoraShower:LArPandoraShowerCreation         19.9341       26.6802       40.1216       24.5466       6.41263        10     
reco:pandoracalo:Calorimetry                        9.07845       12.3429       17.3659       11.1017       3.04971        10     
reco:pandoracalonosce:Calorimetry                   7.2462        11.8691       16.991        11.8038       2.8641         10     
reco:pandorapid:Chi2ParticleID                     0.0111724     0.0164685     0.0233109     0.0153696    0.00382291       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.103768      0.135543      0.179481      0.138152      0.0243235       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.0111462     0.0157566     0.022215      0.0158207    0.00308959       10     
reco:pandoraShowercalo:ShowerCalorimetry            17.2481       30.9163       43.9197       30.9636       7.49452        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       16.409        30.7621       47.2787       30.6845       8.20853        10     
reco:emtrkmichelid:EmTrackMichelId                  384.651       462.586       550.341       441.866       53.1585        10     
reco:crttag:SingleCRTMatchingProducer              0.0305209     0.0530887     0.0798102     0.0494621     0.0173121       10     
reco:crtreco:TwoCRTMatchingProducer                0.0282362     0.0483054     0.076821      0.046026      0.0142403       10     
reco:anodepiercerst0:T0RecoAnodePiercers             1.522        1.76393       2.12279       1.72002      0.190671        10     
reco:pandora2Track:LArPandoraTrackCreation          49.7675       63.5792       83.1732       59.524        11.3747        10     
reco:pandora2calo:Calorimetry                       23.2547       37.718        56.0111       36.9895       8.2616         10     
reco:pandora2calonosce:Calorimetry                  19.4996       34.7985       58.1265       32.2965       10.5787        10     
reco:pandora2pid:Chi2ParticleID                    0.0233647     0.0342234     0.0443336     0.0328482    0.00664743       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.137733      0.168409      0.234103      0.155727      0.0295699       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.026089      0.0338113     0.0428116     0.0334683    0.00535928       10     
reco:pandora2Shower:LArPandoraShowerCreation        15.0964       21.8652       28.696        21.0846       4.84724        10     
reco:pandora2Showercalo:ShowerCalorimetry           19.7914       33.2369       53.5571        32.18        9.66022        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      12.8827       27.9642       44.7169       26.3248       9.37171        10     
[art]:TriggerResults:TriggerResultInserter        6.0033e-05    0.000106933   0.00027611    8.3141e-05    6.28973e-05      10     
end_path:out1:RootOutput                           7.394e-06    1.40024e-05   2.4637e-05    1.1697e-05    5.55783e-06      10     
end_path:out1:RootOutput(write)                     4.30521       5.55387       7.66896       5.33448       1.00218        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5202.32 MB
  Peak resident set size usage (VmHWM): 4361.49 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.22 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.51 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.16 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   6.34 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.24 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.16 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10393168
-rw-r--r-- 1 dune01 dune01 2444652884 Aug  3 13:16 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2325532721 Aug  3 13:11 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2129624935 Aug  3 16:21 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1875435679 Aug  3 09:24 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1791937113 Aug  3 08:43 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   74787810 Aug  3 16:08 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     242982 Aug  3 16:21 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Aug  3 16:21 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Aug  3 16:21 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Aug  3 16:21 time.db
-rw-r--r-- 1 dune01 dune01        519 Aug  3 13:16 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Aug  3 09:24 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Aug  3 16:21 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Aug  3 08:31 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Aug  3 16:21 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Aug  3 13:37 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Aug  3 13:16 debugprod.log
justIN time: 2025-08-04 21:36:23 UTC       justIN version: 01.04.00