justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 238693.0@dunegpschedd01.fnal.gov

Jobsub ID238693.0@dunegpschedd01.fnal.gov
Workflow ID9364
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-10-26 21:35:09
SiteUK_Sheffield
EntryDUNE_UK_Sheffield_lcgce2
Last heartbeat2025-10-27 02:49:27
From worker nodeHostnamewn016.hep
cpuinfoIntel(R) Core(TM) i7-5960X CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-26 21:47:39
Input filesmonte-carlo-009364-000001
JobscriptExit code0
Real time4h (16120s)
CPU time4h (15772s = 97%)
Max RSS bytes4554305536 (4343 MiB)
Outputting started2025-10-27 02:16:20
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09364/1/001/reco_stage1.root
Finished2025-10-27 02:49:27
Saved logsjustin-logs:238693.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

lBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 140617 traces, 0 summary
Retagger: tagging trace set: gauss with 110635 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 110635 traces tagged "gauss"
FrameSaver: q=3.81621e+06 n=1991655 tag=gauss
wclsFrameSaver: saving 140617 traces tagged "wiener"
FrameSaver: q=4.0193e+06 n=1785483 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 49797 XUs and 45073 XVs -> 814 XUVs
C:0 T:1 138258 XUs and 118179 XVs -> 19484 XUVs
C:0 T:2 77252 XUs and 137918 XVs -> 4717 XUVs
C:0 T:3 37097 XUs and 53645 XVs -> 343 XUVs
C:0 T:4 12186 XUs and 8240 XVs -> 233 XUVs
C:0 T:5 100488 XUs and 65582 XVs -> 21153 XUVs
C:0 T:6 67348 XUs and 99205 XVs -> 3016 XUVs
C:0 T:7 27522 XUs and 43649 XVs -> 205 XUVs
C:0 T:8 6443 XUs and 6293 XVs -> 84 XUVs
C:0 T:9 30444 XUs and 37601 XVs -> 1735 XUVs
C:0 T:10 63947 XUs and 44581 XVs -> 7327 XUVs
C:0 T:11 387 XUs and 262 XVs -> 32 XUVs
59143 XUVs total
12426 collection wire objects
56294 potential space points
Neighbour search...
14132792 tests to find 5411522 neighbours
Iterating with no regularization...
Begin: 6.36827e+07
0 5.50009e+07
1 5.42842e+07
2 5.41788e+07
3 5.41467e+07
Now with regularization...
Begin: 4.38002e+07
0 4.37472e+07
1 4.37067e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410134469
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410134469
fLifetime: 30000 [us]
27-Oct-2025 02:16:18 GMT  Closed output file "reco_stage1.root"
27-Oct-2025 02:16:18 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          524.125       685.404       814.309       688.632       91.9828        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00107387    0.00424173    0.00681083    0.00435283    0.00178518       10     
reco:ophit:OpHitFinder                             0.345153       0.4235        0.48527      0.421671      0.0385133       10     
reco:opflash:OpFlashFinder                         0.168393      0.256106      0.310308      0.250691      0.0425724       10     
reco:opslicer:OpSlicer                             0.907829       1.12834       1.29092       1.14611      0.125603        10     
reco:caldata:DataPrepModule                         7.94328       8.22448       8.6941        8.16796      0.203081        10     
reco:wclsdatasp:WireCellToolkit                     56.0535       70.0694       75.9639       73.4163       7.12235        10     
reco:gaushit:GausHitFinder                          1.87268       2.38623       2.92769       2.37391      0.323649        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000626706   0.000822404   0.00112945    0.000809228   0.000127517      10     
reco:reco3d:SpacePointSolver                        9.63486       24.0685       37.1693       24.4145       8.31495        10     
reco:hitpdune:DisambigFromSpacePoints               3.24446       4.85079       6.65334       4.78419       1.05228        10     
reco:pandora:StandardPandora                        47.2863       70.8654       108.081       69.8417       18.107         10     
reco:pandoraWriter:StandardPandora                 0.543896      0.690526      0.826267      0.720477      0.0853657       10     
reco:pandoraTrack:LArPandoraTrackCreation           5.5971        8.6599        11.4517       9.07801       1.66173        10     
reco:pandoraShower:LArPandoraShowerCreation         4.96125       8.58585       13.4952       8.26508       2.45662        10     
reco:pandoracalo:Calorimetry                        3.09797       4.45005       5.56858       4.70257      0.756574        10     
reco:pandoracalonosce:Calorimetry                   2.7894        4.06898       5.16857       4.30007      0.711558        10     
reco:pandorapid:Chi2ParticleID                    0.00798114     0.0107632     0.0126954     0.011157     0.00128716       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0884257      0.10249      0.121353      0.102197      0.0108743       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00666544     0.0106403     0.0130776     0.0108418    0.00162465       10     
reco:pandoraShowercalo:ShowerCalorimetry            5.43679       9.25869       13.7429       8.95393       2.5643         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       5.3306        8.98197       13.4121       8.69217       2.43469        10     
reco:emtrkmichelid:EmTrackMichelId                  289.596       361.367       439.508       359.397       42.3894        10     
reco:crttag:SingleCRTMatchingProducer              0.0147261     0.0216931     0.033071      0.020548     0.00470062       10     
reco:crtreco:TwoCRTMatchingProducer                0.0123778     0.0182481     0.0278152     0.0169955    0.00408075       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.62678       2.18974       2.97895       2.07646      0.355135        10     
reco:pandora2Track:LArPandoraTrackCreation          16.8011       30.2898       41.2199       30.4263       7.94614        10     
reco:pandora2calo:Calorimetry                       8.18134       13.8361       18.0585       13.9736       3.13266        10     
reco:pandora2calonosce:Calorimetry                  7.70248       12.9611       16.8778       12.7036       2.92578        10     
reco:pandora2pid:Chi2ParticleID                    0.0224324     0.0280307     0.0318141     0.0281387    0.00303549       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.112431      0.137336      0.172785      0.134735      0.0185167       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0226267     0.0286714     0.033982      0.0276386     0.0036044       10     
reco:pandora2Shower:LArPandoraShowerCreation        5.88102       9.92255       14.4645       9.46355       2.52426        10     
reco:pandora2Showercalo:ShowerCalorimetry           7.16986       11.8556       15.5123       11.647        2.63783        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      6.63809       10.9896       14.0937       10.8803       2.43703        10     
[art]:TriggerResults:TriggerResultInserter        3.9199e-05    5.31058e-05   0.000147074   4.3048e-05    3.13589e-05      10     
end_path:out1:RootOutput                           5.303e-06    6.2667e-06    1.1389e-05    5.7715e-06    1.72576e-06      10     
end_path:out1:RootOutput(write)                     3.50842       4.55109       5.74623       4.4988       0.612676        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5359.33 MB
  Peak resident set size usage (VmHWM): 4554.31 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.31 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.42 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.27 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11145336
-rw-r--r--. 1 dune004 dune 2578835017 Oct 27 00:19 detsim_stage2.root
-rw-r--r--. 1 dune004 dune 2442417457 Oct 27 00:16 detsim_stage1.root
-rw-r--r--. 1 dune004 dune 2303427250 Oct 27 02:16 reco_stage1.root
-rw-r--r--. 1 dune004 dune 2047575853 Oct 26 22:12 g4_stage2.root
-rw-r--r--. 1 dune004 dune 1956323906 Oct 26 21:55 g4_stage1.root
-rw-r--r--. 1 dune004 dune   83558770 Oct 27 02:08 Pandora_Events.pndr
-rw-r--r--. 1 dune004 dune     242908 Oct 27 02:16 jobscript.log
-rw-r--r--. 1 dune004 dune     217088 Oct 27 02:16 mem.db
-rw-r--r--. 1 dune004 dune      62376 Oct 27 02:16 Pandora_Geometry.xml
-rw-r--r--. 1 dune004 dune      36864 Oct 27 02:16 time.db
-rw-r--r--. 1 dune004 dune        519 Oct 27 00:19 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune004 dune        519 Oct 26 22:12 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune004 dune        519 Oct 27 02:16 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune004 dune         52 Oct 26 21:47 all-input-dids.txt
-rw-r--r--. 1 dune004 dune          7 Oct 27 02:16 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune          0 Oct 27 00:31 ProtoDUNE.log
-rw-r--r--. 1 dune004 dune          0 Oct 27 00:19 debugprod.log
justIN time: 2025-11-03 18:41:03 UTC       justIN version: 01.05.01