justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 232697.0@dunegpschedd02.fnal.gov

Jobsub ID232697.0@dunegpschedd02.fnal.gov
Workflow ID9323
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-10-24 18:49:24
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-10-24 23:17:09
From worker nodeHostnamewn1905240.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-24 18:50:08
Input filesmonte-carlo-009323-000001
JobscriptExit code0
Real time4h (15928s)
CPU time4h (14920s = 93%)
Max RSS bytes4141563904 (3949 MiB)
Outputting started2025-10-24 23:15:38
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09323/1/001/reco_stage1.root
Finished2025-10-24 23:17:09
Saved logsjustin-logs:232697.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

BadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 124007 traces, 0 summary
Retagger: tagging trace set: gauss with 96112 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 96112 traces tagged "gauss"
FrameSaver: q=3.40317e+06 n=1721260 tag=gauss
wclsFrameSaver: saving 124007 traces tagged "wiener"
FrameSaver: q=3.60485e+06 n=1531637 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 51898 XUs and 52765 XVs -> 1150 XUVs
C:0 T:1 128959 XUs and 117383 XVs -> 8272 XUVs
C:0 T:2 82358 XUs and 82817 XVs -> 3258 XUVs
C:0 T:3 46080 XUs and 33497 XVs -> 239 XUVs
C:0 T:4 12834 XUs and 17666 XVs -> 222 XUVs
C:0 T:5 54673 XUs and 55766 XVs -> 7087 XUVs
C:0 T:6 102623 XUs and 88132 XVs -> 4458 XUVs
C:0 T:7 41232 XUs and 32403 XVs -> 254 XUVs
C:0 T:8 5708 XUs and 3818 XVs -> 40 XUVs
C:0 T:9 73226 XUs and 43148 XVs -> 9738 XUVs
C:0 T:10 21931 XUs and 12315 XVs -> 1980 XUVs
C:0 T:11 116 XUs and 64 XVs -> 11 XUVs
36709 XUVs total
11657 collection wire objects
33275 potential space points
Neighbour search...
3846555 tests to find 1684992 neighbours
Iterating with no regularization...
Begin: 1.00218e+08
0 8.01948e+07
1 7.91012e+07
2 7.89842e+07
3 7.89521e+07
Now with regularization...
Begin: 6.67555e+07
0 6.67323e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 410091915
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410091915
fLifetime: 30000 [us]
25-Oct-2025 00:15:35 BST  Closed output file "reco_stage1.root"
25-Oct-2025 00:15:35 BST  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          371.173       516.226       680.093       487.231       120.341        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00145209    0.00495256     0.0069467    0.00537661    0.00166752       10     
reco:ophit:OpHitFinder                             0.256513      0.366272      0.469371       0.37702      0.0569932       10     
reco:opflash:OpFlashFinder                         0.159982      0.207086      0.288287      0.206412      0.0365163       10     
reco:opslicer:OpSlicer                             0.573281      0.853226       1.04127      0.900449      0.148337        10     
reco:caldata:DataPrepModule                         7.24733       7.46571       8.00919       7.4393       0.201101        10     
reco:wclsdatasp:WireCellToolkit                     41.1743       46.4641       51.9933       46.1283       3.02157        10     
reco:gaushit:GausHitFinder                          1.38125       1.7627        2.42444       1.72152      0.327051        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000608544   0.000777982   0.00123333    0.00071303    0.000194152      10     
reco:reco3d:SpacePointSolver                        13.7633       34.1956       74.0163       25.4162       19.6386        10     
reco:hitpdune:DisambigFromSpacePoints               1.25891       2.38932       4.08792       2.09156      0.819061        10     
reco:pandora:StandardPandora                        27.5198       58.0758       109.299       50.3385       25.0589        10     
reco:pandoraWriter:StandardPandora                  0.35758      0.677297       1.09867      0.668411      0.183081        10     
reco:pandoraTrack:LArPandoraTrackCreation           5.29474       8.17232       13.6969       7.57022       2.49664        10     
reco:pandoraShower:LArPandoraShowerCreation         4.82937       9.46753       15.7462       8.43618       3.97866        10     
reco:pandoracalo:Calorimetry                        2.89129       4.59854       6.9745        4.49355       1.37088        10     
reco:pandoracalonosce:Calorimetry                   2.59249       4.36972       6.87838       4.06988       1.31568        10     
reco:pandorapid:Chi2ParticleID                    0.00686455    0.00946749     0.0150046    0.00884415    0.00218071       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0599314     0.0805114     0.122511      0.0778948     0.0178759       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00694908    0.00900337     0.0125279    0.00861724    0.00161243       10     
reco:pandoraShowercalo:ShowerCalorimetry            4.09009       9.11689       14.8001       8.1318        3.78108        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.83176       8.83544       14.9231       8.62396       4.03214        10     
reco:emtrkmichelid:EmTrackMichelId                  193.395       241.999       303.369       232.731       41.3957        10     
reco:crttag:SingleCRTMatchingProducer              0.0157738     0.0242432     0.0349048     0.024243     0.00566721       10     
reco:crtreco:TwoCRTMatchingProducer               0.00847764     0.0151304     0.0231488     0.0144689     0.0049351       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.816124       1.02124       1.23184       1.03562      0.134888        10     
reco:pandora2Track:LArPandoraTrackCreation          14.2569       23.8174       37.0735       21.0997       8.26877        10     
reco:pandora2calo:Calorimetry                       6.43053       11.7671       18.1298       10.7308       4.5273         10     
reco:pandora2calonosce:Calorimetry                  6.36414       11.1861       17.4758       10.1586       4.33678        10     
reco:pandora2pid:Chi2ParticleID                    0.015356      0.020257      0.0277305     0.0182955    0.00417244       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0761354     0.105282      0.189109      0.0955498     0.0320287       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0151343     0.0196627     0.0277564     0.0177515    0.00404334       10     
reco:pandora2Shower:LArPandoraShowerCreation        5.54332       8.10733       12.1993       7.34551       2.3465         10     
reco:pandora2Showercalo:ShowerCalorimetry           5.18241       9.02213       14.9384       8.42461       3.00174        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      5.31318       7.91912       12.3118       7.26029       2.53296        10     
[art]:TriggerResults:TriggerResultInserter        6.2788e-05    9.85092e-05   0.000235183   8.5391e-05    4.63874e-05      10     
end_path:out1:RootOutput                          1.3746e-05    1.64468e-05   2.6901e-05    1.52435e-05   3.66746e-06      10     
end_path:out1:RootOutput(write)                     3.21739       3.89169       5.41323       3.53896      0.711901        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4974.19 MB
  Peak resident set size usage (VmHWM): 4141.56 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.35 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.65 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.20 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10070104
-rw-r--r-- 1 dune01 dune01 2376836767 Oct 24 22:47 detsim_stage2.root
-rw-r--r-- 1 dune01 dune01 2255494202 Oct 24 22:44 detsim_stage1.root
-rw-r--r-- 1 dune01 dune01 2065471882 Oct 25 00:15 reco_stage1.root
-rw-r--r-- 1 dune01 dune01 1813072755 Oct 24 20:22 g4_stage2.root
-rw-r--r-- 1 dune01 dune01 1724264544 Oct 24 19:58 g4_stage1.root
-rw-r--r-- 1 dune01 dune01   76014530 Oct 25 00:09 Pandora_Events.pndr
-rw-r--r-- 1 dune01 dune01     243299 Oct 25 00:15 jobscript.log
-rw-r--r-- 1 dune01 dune01     217088 Oct 25 00:15 mem.db
-rw-r--r-- 1 dune01 dune01      62376 Oct 25 00:15 Pandora_Geometry.xml
-rw-r--r-- 1 dune01 dune01      36864 Oct 25 00:15 time.db
-rw-r--r-- 1 dune01 dune01        519 Oct 24 22:47 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Oct 24 20:22 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune01 dune01        519 Oct 25 00:15 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune01 dune01         52 Oct 24 19:50 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01          7 Oct 25 00:15 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01          0 Oct 24 22:57 ProtoDUNE.log
-rw-r--r-- 1 dune01 dune01          0 Oct 24 22:47 debugprod.log
justIN time: 2025-11-03 18:40:58 UTC       justIN version: 01.05.01