justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 264621.0@dunegpschedd02.fnal.gov

Jobsub ID264621.0@dunegpschedd02.fnal.gov
Workflow ID11366
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-27 22:55:13
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2025-12-28 02:48:50
From worker nodeHostnamehnode73.pic.es
cpuinfoAMD EPYC 7443P 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-27 23:04:14
Input filesmonte-carlo-011366-000001
JobscriptExit code0
Real time3h (13401s)
CPU time3h (13064s = 97%)
Max RSS bytes4893450240 (4666 MiB)
Outputting started2025-12-28 02:47:36
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11366/1/001/reco_stage1.root
Finished2025-12-28 02:48:50
Saved logsjustin-logs:264621.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 150340 traces, 0 summary
Retagger: tagging trace set: gauss with 119906 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 119906 traces tagged "gauss"
FrameSaver: q=4.18513e+06 n=2240389 tag=gauss
wclsFrameSaver: saving 150340 traces tagged "wiener"
FrameSaver: q=4.4092e+06 n=2024388 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 25004 XUs and 24205 XVs -> 233 XUVs
C:0 T:1 40760 XUs and 34953 XVs -> 1922 XUVs
C:0 T:2 178194 XUs and 226637 XVs -> 17196 XUVs
C:0 T:3 81623 XUs and 75704 XVs -> 857 XUVs
C:0 T:4 17433 XUs and 21240 XVs -> 177 XUVs
C:0 T:5 75765 XUs and 86413 XVs -> 11650 XUVs
C:0 T:6 159345 XUs and 164214 XVs -> 6454 XUVs
C:0 T:7 60828 XUs and 60390 XVs -> 563 XUVs
C:0 T:8 9896 XUs and 11581 XVs -> 164 XUVs
C:0 T:9 64898 XUs and 91576 XVs -> 6483 XUVs
C:0 T:10 43316 XUs and 19826 XVs -> 5841 XUVs
C:0 T:11 150 XUs and 78 XVs -> 11 XUVs
51551 XUVs total
14218 collection wire objects
47947 potential space points
Neighbour search...
7213131 tests to find 3312242 neighbours
Iterating with no regularization...
Begin: 8.32429e+07
0 7.00359e+07
1 6.91264e+07
2 6.90146e+07
3 6.898e+07
Now with regularization...
Begin: 5.68887e+07
0 5.68349e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411382712
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411382712
fLifetime: 30000 [us]
28-Dec-2025 03:47:33 CET  Closed output file "reco_stage1.root"
28-Dec-2025 03:47:33 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          268.682       434.446       520.959       464.428       79.3151        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000741876   0.00390897    0.00656871    0.00394546    0.00170761       10     
reco:ophit:OpHitFinder                             0.245878      0.331987      0.432038      0.318357      0.0702648       10     
reco:opflash:OpFlashFinder                         0.176621      0.233165      0.329953      0.231917      0.0482207       10     
reco:opslicer:OpSlicer                             0.559594      0.827699       1.09826      0.837157      0.187055        10     
reco:caldata:DataPrepModule                         3.8516        5.2209        6.75612       5.0867        1.05194        10     
reco:wclsdatasp:WireCellToolkit                     29.6102       34.2967       38.9487       34.1991       3.36741        10     
reco:gaushit:GausHitFinder                         0.974444       1.49128       2.12899       1.31154      0.396345        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000343498   0.000574758   0.000799489   0.000580963   0.000131696      10     
reco:reco3d:SpacePointSolver                        10.2838       23.4783       39.0413       20.3654       9.70054        10     
reco:hitpdune:DisambigFromSpacePoints               1.46329       2.30676       3.38058       2.14301      0.608819        10     
reco:pandora:StandardPandora                        17.4251       53.0699       77.7793       56.9479       17.6223        10     
reco:pandoraWriter:StandardPandora                 0.264168      0.549135       0.71527      0.597108      0.144337        10     
reco:pandoraTrack:LArPandoraTrackCreation           3.41138       6.45358       9.23358       6.63802       1.80164        10     
reco:pandoraShower:LArPandoraShowerCreation         2.24297       7.14119       11.4256       6.63031       2.54278        10     
reco:pandoracalo:Calorimetry                        1.30532       3.3317        5.01348       3.32509       1.01844        10     
reco:pandoracalonosce:Calorimetry                   1.70999       3.02601       4.30999       3.16632      0.695625        10     
reco:pandorapid:Chi2ParticleID                    0.00409292    0.00669796    0.00977779    0.00702157    0.00196437       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0342786     0.060113      0.0823391     0.0587636     0.0183852       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00416852    0.00659638    0.00886441     0.0069052    0.00179157       10     
reco:pandoraShowercalo:ShowerCalorimetry            2.58363       6.57873       10.5868       6.46635       1.91935        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.49617       6.69049       10.1898       6.26595       1.93822        10     
reco:emtrkmichelid:EmTrackMichelId                  141.431       215.227       263.229       222.77        37.2293        10     
reco:crttag:SingleCRTMatchingProducer              0.0102454     0.0154751     0.021939      0.0153569     0.0036798       10     
reco:crtreco:TwoCRTMatchingProducer               0.00787478     0.0131327     0.0197333     0.011757     0.00384336       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.808058       1.03502       1.33475      0.986433      0.160036        10     
reco:pandora2Track:LArPandoraTrackCreation          8.74224       18.7659       28.2598       16.1534       6.01972        10     
reco:pandora2calo:Calorimetry                       3.70833       9.33719       14.4142       8.61843       3.00512        10     
reco:pandora2calonosce:Calorimetry                  3.3468        8.96733       13.2122       8.97701       2.72575        10     
reco:pandora2pid:Chi2ParticleID                    0.0131002     0.0205554     0.0245598     0.0206905    0.00298236       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0592127     0.0961069     0.120127       0.10353      0.0194396       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0111725     0.0195327     0.0238737     0.0205976    0.00391749       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.20706       6.5512        8.91366       6.84222       1.92335        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.41254       8.1168        11.9481       8.09427       2.6352         10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.65102       7.74457       10.5106       7.47173       2.05625        10     
[art]:TriggerResults:TriggerResultInserter        3.1401e-05    4.55912e-05   0.000116103   3.7386e-05    2.39522e-05      10     
end_path:out1:RootOutput                           4.88e-06      6.471e-06     1.031e-05     5.97e-06     1.46006e-06      10     
end_path:out1:RootOutput(write)                     2.12382       3.27949       4.40488       3.16875      0.670066        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5704.13 MB
  Peak resident set size usage (VmHWM): 4893.45 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.22 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.31 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.13 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 12134944
-rw-r--r-- 1 dune001 dune 2775982528 Dec 28 02:33 detsim_stage2.root
-rw-r--r-- 1 dune001 dune 2632816230 Dec 28 02:30 detsim_stage1.root
-rw-r--r-- 1 dune001 dune 2514298343 Dec 28 03:47 reco_stage1.root
-rw-r--r-- 1 dune001 dune 2249532397 Dec 28 00:24 g4_stage2.root
-rw-r--r-- 1 dune001 dune 2165362726 Dec 28 00:12 g4_stage1.root
-rw-r--r-- 1 dune001 dune   87553310 Dec 28 03:41 Pandora_Events.pndr
-rw-r--r-- 1 dune001 dune     242949 Dec 28 03:47 jobscript.log
-rw-r--r-- 1 dune001 dune     217088 Dec 28 03:47 mem.db
-rw-r--r-- 1 dune001 dune      62376 Dec 28 03:47 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 dune      36864 Dec 28 03:47 time.db
-rw-r--r-- 1 dune001 dune        519 Dec 28 02:33 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune        519 Dec 28 00:24 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune        519 Dec 28 03:47 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune001 dune         52 Dec 28 00:04 all-input-dids.txt
-rw-r--r-- 1 dune001 dune          7 Dec 28 03:47 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune          0 Dec 28 02:40 ProtoDUNE.log
-rw-r--r-- 1 dune001 dune          0 Dec 28 02:33 debugprod.log
justIN time: 2026-02-07 03:36:10 UTC       justIN version: 01.06.00