justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 264592.0@dunegpschedd02.fnal.gov

Jobsub ID264592.0@dunegpschedd02.fnal.gov
Workflow ID11359
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-27 16:54:55
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce16-multicore
Last heartbeat2025-12-27 21:13:19
From worker nodeHostnamehnode69.pic.es
cpuinfoAMD EPYC 7402P 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-27 16:58:50
Input filesmonte-carlo-011359-000001
JobscriptExit code0
Real time4h (15198s)
CPU time3h (13411s = 88%)
Max RSS bytes4596445184 (4383 MiB)
Outputting started2025-12-27 21:12:09
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11359/1/001/reco_stage1.root
Finished2025-12-27 21:13:19
Saved logsjustin-logs:264592.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 154764 traces, 0 summary
Retagger: tagging trace set: gauss with 119799 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 119799 traces tagged "gauss"
FrameSaver: q=4.21689e+06 n=2149873 tag=gauss
wclsFrameSaver: saving 154764 traces tagged "wiener"
FrameSaver: q=4.4422e+06 n=1918680 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 60000 XUs and 55951 XVs -> 1133 XUVs
C:0 T:1 118692 XUs and 110232 XVs -> 8083 XUVs
C:0 T:2 111155 XUs and 158753 XVs -> 5423 XUVs
C:0 T:3 56421 XUs and 62198 XVs -> 393 XUVs
C:0 T:4 21705 XUs and 18977 XVs -> 239 XUVs
C:0 T:5 91927 XUs and 64897 XVs -> 9191 XUVs
C:0 T:6 40934 XUs and 34911 XVs -> 1290 XUVs
C:0 T:7 20884 XUs and 17049 XVs -> 71 XUVs
C:0 T:8 10215 XUs and 7549 XVs -> 123 XUVs
C:0 T:9 74772 XUs and 68052 XVs -> 5788 XUVs
C:0 T:10 39238 XUs and 28706 XVs -> 3518 XUVs
C:0 T:11 141 XUs and 112 XVs -> 14 XUVs
35266 XUVs total
13541 collection wire objects
32309 potential space points
Neighbour search...
1611819 tests to find 883746 neighbours
Iterating with no regularization...
Begin: 8.19649e+07
0 7.08111e+07
1 6.99798e+07
2 6.98691e+07
3 6.98375e+07
Now with regularization...
Begin: 5.76365e+07
0 5.76122e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411377787
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411377787
fLifetime: 30000 [us]
27-Dec-2025 22:12:06 CET  Closed output file "reco_stage1.root"
27-Dec-2025 22:12:06 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          368.809       459.619       552.327       443.408       61.4866        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000647884   0.00306408    0.00458953    0.00325962    0.00131482       10     
reco:ophit:OpHitFinder                             0.264974      0.309099      0.418251      0.289125      0.047146        10     
reco:opflash:OpFlashFinder                         0.158072      0.222951      0.266801      0.224411      0.0341448       10     
reco:opslicer:OpSlicer                             0.625131      0.755644       1.19987       0.71055      0.162665        10     
reco:caldata:DataPrepModule                         4.31839       4.94122       7.04113       4.62322      0.836727        10     
reco:wclsdatasp:WireCellToolkit                     28.5362       34.0072       41.5377       34.5742       3.76474        10     
reco:gaushit:GausHitFinder                          1.01581       1.48682       2.2806        1.3769       0.407957        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000473617   0.00069479    0.000960335   0.000613573   0.000175424      10     
reco:reco3d:SpacePointSolver                        11.3677       18.9998       30.7544       15.7956       6.89721        10     
reco:hitpdune:DisambigFromSpacePoints               1.20729       2.01554       3.23199       1.93076      0.579938        10     
reco:pandora:StandardPandora                        26.7171       44.709        74.8081       43.8887       15.195         10     
reco:pandoraWriter:StandardPandora                 0.389043      0.453904      0.579019      0.446907      0.0549125       10     
reco:pandoraTrack:LArPandoraTrackCreation           3.79934       5.18294       8.33046       4.76814       1.39532        10     
reco:pandoraShower:LArPandoraShowerCreation         3.91644       6.65762       11.6535       5.61154       2.35863        10     
reco:pandoracalo:Calorimetry                        1.71033       2.36064       3.73762       2.21375      0.615563        10     
reco:pandoracalonosce:Calorimetry                   1.48802       2.34264       3.21613       2.45056      0.517033        10     
reco:pandorapid:Chi2ParticleID                     0.0048987    0.00649799    0.00868997    0.00626372    0.00121285       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0468926     0.0586911     0.077832      0.0556274    0.00983254       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00450827    0.00605692    0.00860028     0.0056794    0.00130839       10     
reco:pandoraShowercalo:ShowerCalorimetry            3.23551       5.51134       8.75243       4.78205       1.74189        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.33954       5.65807       10.7267       4.63479       2.31247        10     
reco:emtrkmichelid:EmTrackMichelId                  213.271       264.401       323.278       269.438       32.2389        10     
reco:crttag:SingleCRTMatchingProducer             0.00955755     0.0146432     0.0197342     0.0145748    0.00318275       10     
reco:crtreco:TwoCRTMatchingProducer               0.00827358     0.0119239     0.0187606     0.0104707    0.00325831       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.790559       1.05949       1.43239       1.02074      0.181912        10     
reco:pandora2Track:LArPandoraTrackCreation          13.5025       18.6234       25.2782       17.3676       3.85988        10     
reco:pandora2calo:Calorimetry                       5.10377       7.77608       10.052        7.83557       1.68641        10     
reco:pandora2calonosce:Calorimetry                  4.65603       7.74184       9.9831        7.8802        1.42861        10     
reco:pandora2pid:Chi2ParticleID                    0.0128034     0.0189423     0.0249905     0.0189386     0.0039005       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0657127     0.0872177     0.114847      0.0846154     0.0185515       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0127636     0.0180065     0.0246809     0.0167979    0.00407624       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.60736       7.17251       10.5018       6.66363       1.74202        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.36916       7.16631       9.40995       7.06818       1.53264        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.99173       6.22727       7.46886       6.46592       1.0664         10     
[art]:TriggerResults:TriggerResultInserter        2.0531e-05    3.43571e-05   8.8753e-05    2.7896e-05    1.87554e-05      10     
end_path:out1:RootOutput                           4.69e-06     5.6632e-06     9.55e-06     5.2555e-06    1.34404e-06      10     
end_path:out1:RootOutput(write)                     2.86052       3.46865       4.12448       3.41689      0.419294        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5408.15 MB
  Peak resident set size usage (VmHWM): 4596.45 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.20 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.15 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.12 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11436256
-rw-r--r-- 1 dune001 dune 2645027986 Dec 27 20:52 detsim_stage2.root
-rw-r--r-- 1 dune001 dune 2506787704 Dec 27 20:50 detsim_stage1.root
-rw-r--r-- 1 dune001 dune 2392009908 Dec 27 22:12 reco_stage1.root
-rw-r--r-- 1 dune001 dune 2086238005 Dec 27 18:42 g4_stage2.root
-rw-r--r-- 1 dune001 dune 1991034740 Dec 27 18:07 g4_stage1.root
-rw-r--r-- 1 dune001 dune   89001240 Dec 27 22:06 Pandora_Events.pndr
-rw-r--r-- 1 dune001 dune     242966 Dec 27 22:12 jobscript.log
-rw-r--r-- 1 dune001 dune     217088 Dec 27 22:12 mem.db
-rw-r--r-- 1 dune001 dune      62376 Dec 27 22:12 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 dune      36864 Dec 27 22:12 time.db
-rw-r--r-- 1 dune001 dune        519 Dec 27 20:52 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune        519 Dec 27 18:42 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune        519 Dec 27 22:12 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune001 dune         52 Dec 27 17:58 all-input-dids.txt
-rw-r--r-- 1 dune001 dune          7 Dec 27 22:12 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune          0 Dec 27 20:59 ProtoDUNE.log
-rw-r--r-- 1 dune001 dune          0 Dec 27 20:53 debugprod.log
justIN time: 2026-02-07 03:34:54 UTC       justIN version: 01.06.00