justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 271064.1@dunegpschedd02.fnal.gov

Jobsub ID271064.1@dunegpschedd02.fnal.gov
Workflow ID11792
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-13 06:13:01
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2026-01-13 11:06:14
From worker nodeHostnamehnode62.pic.es
cpuinfoAMD EPYC 7402P 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-13 06:13:42
Input filesmonte-carlo-011792-000001
JobscriptExit code0
Real time4h (17481s)
CPU time3h (14108s = 80%)
Max RSS bytes4315750400 (4115 MiB)
Outputting started2026-01-13 11:05:04
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11792/1/001/reco_stage1.root
Finished2026-01-13 11:06:14
Saved logsjustin-logs:271064.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Running tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 145295 traces, 0 summary
Retagger: tagging trace set: gauss with 112969 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 112969 traces tagged "gauss"
FrameSaver: q=4.06061e+06 n=2043655 tag=gauss
wclsFrameSaver: saving 145295 traces tagged "wiener"
FrameSaver: q=4.29744e+06 n=1833267 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 41371 XUs and 34301 XVs -> 949 XUVs
C:0 T:1 107476 XUs and 98231 XVs -> 11747 XUVs
C:0 T:2 90594 XUs and 74762 XVs -> 2752 XUVs
C:0 T:3 43183 XUs and 29911 XVs -> 202 XUVs
C:0 T:4 22081 XUs and 21612 XVs -> 400 XUVs
C:0 T:5 88052 XUs and 78037 XVs -> 7430 XUVs
C:0 T:6 137263 XUs and 125798 XVs -> 6902 XUVs
C:0 T:7 48312 XUs and 48085 XVs -> 380 XUVs
C:0 T:8 10805 XUs and 15994 XVs -> 1749 XUVs
C:0 T:9 53581 XUs and 70301 XVs -> 5389 XUVs
C:0 T:10 46589 XUs and 29475 XVs -> 4798 XUVs
C:0 T:11 327 XUs and 264 XVs -> 44 XUVs
42742 XUVs total
13543 collection wire objects
38835 potential space points
Neighbour search...
3396575 tests to find 1770482 neighbours
Iterating with no regularization...
Begin: 8.75074e+07
0 7.30624e+07
1 7.2171e+07
2 7.20245e+07
3 7.19771e+07
Now with regularization...
Begin: 6.02558e+07
0 6.01994e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411710763
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411710763
fLifetime: 30000 [us]
13-Jan-2026 12:05:02 CET  Closed output file "reco_stage1.root"
13-Jan-2026 12:05:02 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          342.557       466.01        567.231       478.399       70.6451        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000790208   0.00362298    0.00582627    0.00388387     0.0017153       10     
reco:ophit:OpHitFinder                             0.258746       0.3066       0.412962      0.288315      0.0514225       10     
reco:opflash:OpFlashFinder                         0.159038      0.214458      0.260143      0.202432      0.0329481       10     
reco:opslicer:OpSlicer                             0.624529      0.784002       1.24382      0.663083      0.221961        10     
reco:caldata:DataPrepModule                         4.46694       4.95114       6.47481       4.62065      0.702311        10     
reco:wclsdatasp:WireCellToolkit                     32.9254       36.9973       43.6397       36.3683       2.90499        10     
reco:gaushit:GausHitFinder                         0.998353       1.38958       1.89291       1.36235      0.295746        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00054541    0.000725408   0.000920683   0.000684285   0.000113858      10     
reco:reco3d:SpacePointSolver                        9.91294       17.8702       27.8804       18.4115       5.83863        10     
reco:hitpdune:DisambigFromSpacePoints               1.27541       1.77136       2.27676       1.75078      0.347285        10     
reco:pandora:StandardPandora                        25.702        42.1817       74.1088       38.5117       13.8844        10     
reco:pandoraWriter:StandardPandora                 0.378586      0.492087      0.700558      0.462957      0.0978118       10     
reco:pandoraTrack:LArPandoraTrackCreation           3.68285       5.63384       8.3237        5.84262       1.51375        10     
reco:pandoraShower:LArPandoraShowerCreation         3.20536       6.93781       12.8405       7.11442       2.87612        10     
reco:pandoracalo:Calorimetry                        1.88872       3.08397       7.53566       2.38277       1.55587        10     
reco:pandoracalonosce:Calorimetry                   1.63284       2.5574        4.89803       2.11796      0.932983        10     
reco:pandorapid:Chi2ParticleID                    0.00494159    0.00596357    0.00843939    0.00559488    0.00102203       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0463442      0.05488      0.0897315     0.0506193     0.0122039       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00493628    0.00582896    0.00870997    0.00534537    0.00107861       10     
reco:pandoraShowercalo:ShowerCalorimetry            3.06819       6.91763       14.7019       6.43671        3.266         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.08039       6.34951       10.9701       6.52536       2.21686        10     
reco:emtrkmichelid:EmTrackMichelId                  198.318       263.248       322.558       279.314       39.7742        10     
reco:crttag:SingleCRTMatchingProducer             0.00877805     0.0137434     0.0231534     0.0129841    0.00498404       10     
reco:crtreco:TwoCRTMatchingProducer               0.00767203     0.0117143     0.0225238     0.010098     0.00444786       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.734527      0.926138       1.35756      0.877718      0.173742        10     
reco:pandora2Track:LArPandoraTrackCreation          11.6087       19.6888       30.4314       19.9883       5.39848        10     
reco:pandora2calo:Calorimetry                       5.35099       9.47805       14.0783       9.5192        2.5272         10     
reco:pandora2calonosce:Calorimetry                  5.08722       9.39509       14.5851       9.05034       2.77587        10     
reco:pandora2pid:Chi2ParticleID                    0.0125869     0.0161922     0.024595      0.0144493    0.00397675       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0563843     0.0754603     0.120299      0.0666583     0.0226163       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0125431     0.0159808     0.0225786     0.0148297    0.00364148       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.14544       7.07825       9.50042       6.73994       1.61363        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.55934       7.16556       9.7467        6.7399        1.68126        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      4.25052       6.98017       9.08556       7.73395       1.70117        10     
[art]:TriggerResults:TriggerResultInserter        2.2331e-05    3.70834e-05   9.0003e-05    3.0471e-05    1.95489e-05      10     
end_path:out1:RootOutput                           4.73e-06     5.8653e-06     9.971e-06    5.4855e-06    1.41837e-06      10     
end_path:out1:RootOutput(write)                     2.74089       3.25062       4.50694       3.12273      0.485401        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5127.77 MB
  Peak resident set size usage (VmHWM): 4315.75 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.20 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.14 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.12 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11167188
-rw-r--r-- 1 dune001 dune 2577612296 Jan 13 10:44 detsim_stage2.root
-rw-r--r-- 1 dune001 dune 2440915924 Jan 13 10:42 detsim_stage1.root
-rw-r--r-- 1 dune001 dune 2307860429 Jan 13 12:05 reco_stage1.root
-rw-r--r-- 1 dune001 dune 2056668949 Jan 13 08:01 g4_stage2.root
-rw-r--r-- 1 dune001 dune 1966169201 Jan 13 07:36 g4_stage1.root
-rw-r--r-- 1 dune001 dune   85346050 Jan 13 11:58 Pandora_Events.pndr
-rw-r--r-- 1 dune001 dune     242788 Jan 13 12:05 jobscript.log
-rw-r--r-- 1 dune001 dune     217088 Jan 13 12:05 mem.db
-rw-r--r-- 1 dune001 dune      62376 Jan 13 12:05 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 dune      36864 Jan 13 12:05 time.db
-rw-r--r-- 1 dune001 dune        519 Jan 13 10:44 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune        519 Jan 13 08:01 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune        519 Jan 13 12:05 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune001 dune         52 Jan 13 07:13 all-input-dids.txt
-rw-r--r-- 1 dune001 dune          7 Jan 13 12:05 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune          0 Jan 13 10:51 ProtoDUNE.log
-rw-r--r-- 1 dune001 dune          0 Jan 13 10:44 debugprod.log
justIN time: 2026-02-06 07:39:14 UTC       justIN version: 01.06.00