justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 264532.1@dunegpschedd02.fnal.gov

Jobsub ID264532.1@dunegpschedd02.fnal.gov
Workflow ID11352
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-27 10:06:34
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2025-12-27 13:48:50
From worker nodeHostnamegpu01.pic.es
cpuinfoIntel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-27 10:14:10
Input filesmonte-carlo-011352-000001
JobscriptExit code0
Real time3h (12842s)
CPU time3h (12175s = 94%)
Max RSS bytes4619476992 (4405 MiB)
Outputting started2025-12-27 13:48:14
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11352/1/001/reco_stage1.root
Finished2025-12-27 13:48:50
Saved logsjustin-logs:264532.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

cKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 145591 traces, 0 summary
Retagger: tagging trace set: gauss with 110144 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 110144 traces tagged "gauss"
FrameSaver: q=3.8809e+06 n=1950757 tag=gauss
wclsFrameSaver: saving 145591 traces tagged "wiener"
FrameSaver: q=4.1299e+06 n=1719830 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 31539 XUs and 22650 XVs -> 185 XUVs
C:0 T:1 56695 XUs and 50024 XVs -> 5055 XUVs
C:0 T:2 138049 XUs and 188108 XVs -> 7804 XUVs
C:0 T:3 62570 XUs and 69497 XVs -> 1071 XUVs
C:0 T:4 12460 XUs and 12916 XVs -> 114 XUVs
C:0 T:5 60618 XUs and 157198 XVs -> 8053 XUVs
C:0 T:6 105518 XUs and 101462 XVs -> 5928 XUVs
C:0 T:7 40741 XUs and 38402 XVs -> 360 XUVs
C:0 T:8 10755 XUs and 12945 XVs -> 132 XUVs
C:0 T:9 72514 XUs and 116217 XVs -> 6584 XUVs
C:0 T:10 83090 XUs and 28373 XVs -> 14123 XUVs
C:0 T:11 312 XUs and 129 XVs -> 78 XUVs
49487 XUVs total
13624 collection wire objects
44685 potential space points
Neighbour search...
5916885 tests to find 2594910 neighbours
Iterating with no regularization...
Begin: 1.13636e+08
0 7.77545e+07
1 7.57306e+07
2 7.54776e+07
3 7.5389e+07
4 7.53436e+07
Now with regularization...
Begin: 6.37258e+07
0 6.36972e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411372113
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411372113
fLifetime: 30000 [us]
27-Dec-2025 14:48:11 CET  Closed output file "reco_stage1.root"
27-Dec-2025 14:48:11 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          402.752       518.817       605.909       518.162       55.6959        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000992617   0.00343666    0.00529895    0.00343153    0.00133297       10     
reco:ophit:OpHitFinder                             0.256876      0.301558      0.344376      0.305407      0.0274744       10     
reco:opflash:OpFlashFinder                         0.220665      0.280724      0.371492      0.283936      0.0437042       10     
reco:opslicer:OpSlicer                             0.608609      0.717404      0.852541      0.709324      0.0778186       10     
reco:caldata:DataPrepModule                         5.20993       5.46637       6.15967       5.38517      0.244668        10     
reco:wclsdatasp:WireCellToolkit                     43.9457       47.8066       48.9992       48.5051       1.61972        10     
reco:gaushit:GausHitFinder                          1.43747       1.68861       1.92683       1.69089      0.150393        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000622747   0.000747336   0.00099305    0.000739679   0.000104664      10     
reco:reco3d:SpacePointSolver                        10.0069       17.7319       23.1497       18.3877       3.45475        10     
reco:hitpdune:DisambigFromSpacePoints               1.96476       2.79409       3.84429       2.72015      0.537156        10     
reco:pandora:StandardPandora                        38.2394       57.1353       73.4151       59.6901       10.7313        10     
reco:pandoraWriter:StandardPandora                  0.45321       1.38877       3.45373      0.883785       1.08827        10     
reco:pandoraTrack:LArPandoraTrackCreation           10.1587        12.06        14.5882       11.824        1.33502        10     
reco:pandoraShower:LArPandoraShowerCreation         9.33135       14.1393       17.3945       13.5275       2.59232        10     
reco:pandoracalo:Calorimetry                        4.68398       5.51605       6.68008       5.38647      0.565324        10     
reco:pandoracalonosce:Calorimetry                   4.33115       5.22071       6.36853       5.14188      0.574533        10     
reco:pandorapid:Chi2ParticleID                    0.00601739    0.00690247    0.00814677    0.00673675    0.00063957       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0625643     0.0764696     0.0913097     0.076171     0.00890478       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00592024    0.00657532    0.00749666    0.00649699    0.000464673      10     
reco:pandoraShowercalo:ShowerCalorimetry            8.14787       12.8984       15.7552       12.724        2.4713         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       7.78758       12.437        15.2341       12.1902       2.36236        10     
reco:emtrkmichelid:EmTrackMichelId                  169.203       209.051       240.548       214.488       21.7749        10     
reco:crttag:SingleCRTMatchingProducer              0.0259625     0.0291778     0.0335521     0.0292294    0.00216265       10     
reco:crtreco:TwoCRTMatchingProducer                0.0186932     0.0238513     0.0270659     0.0243635    0.00242567       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.04448       1.45002       1.74755       1.43894      0.214814        10     
reco:pandora2Track:LArPandoraTrackCreation          27.4121       38.2407       47.572        37.2438       6.32766        10     
reco:pandora2calo:Calorimetry                        11.81        16.564        20.4279       16.3069       2.67107        10     
reco:pandora2calonosce:Calorimetry                  10.8496       15.5512       19.1789       15.5698       2.54047        10     
reco:pandora2pid:Chi2ParticleID                    0.0137537     0.0165072     0.0189729     0.0161342     0.0016598       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0784849     0.0951109     0.108749      0.0933042     0.0101817       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0137939     0.0163031      0.01833      0.0156998    0.00166076       10     
reco:pandora2Shower:LArPandoraShowerCreation        8.17607       11.3204       13.6108       11.6132       1.77793        10     
reco:pandora2Showercalo:ShowerCalorimetry           9.77624       12.8131       15.6718       12.8403       1.79828        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      7.93827       10.8578       13.2288       11.1986       1.65187        10     
[art]:TriggerResults:TriggerResultInserter        4.1535e-05    5.95416e-05   0.000131617   5.2728e-05    2.43116e-05      10     
end_path:out1:RootOutput                           8.095e-06    9.3866e-06    1.4022e-05     8.84e-06     1.59812e-06      10     
end_path:out1:RootOutput(write)                     2.39801       4.94501       11.2565       3.64245       2.63763        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5430.85 MB
  Peak resident set size usage (VmHWM): 4619.48 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.21 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.45 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.16 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11372380
-rw-r--r-- 1 dune001 dune 2634169665 Dec 27 13:18 detsim_stage2.root
-rw-r--r-- 1 dune001 dune 2499042595 Dec 27 13:15 detsim_stage1.root
-rw-r--r-- 1 dune001 dune 2371435180 Dec 27 14:48 reco_stage1.root
-rw-r--r-- 1 dune001 dune 2066257401 Dec 27 11:40 g4_stage2.root
-rw-r--r-- 1 dune001 dune 1986471248 Dec 27 11:23 g4_stage1.root
-rw-r--r-- 1 dune001 dune   87294810 Dec 27 14:42 Pandora_Events.pndr
-rw-r--r-- 1 dune001 dune     243046 Dec 27 14:48 jobscript.log
-rw-r--r-- 1 dune001 dune     217088 Dec 27 14:48 mem.db
-rw-r--r-- 1 dune001 dune      62376 Dec 27 14:48 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 dune      36864 Dec 27 14:48 time.db
-rw-r--r-- 1 dune001 dune        519 Dec 27 13:18 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune        519 Dec 27 11:40 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune        519 Dec 27 14:48 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune001 dune         52 Dec 27 11:14 all-input-dids.txt
-rw-r--r-- 1 dune001 dune          7 Dec 27 14:48 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune          0 Dec 27 13:27 ProtoDUNE.log
-rw-r--r-- 1 dune001 dune          0 Dec 27 13:18 debugprod.log
justIN time: 2026-02-07 03:33:50 UTC       justIN version: 01.06.00