justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 265273.0@dunegpschedd02.fnal.gov

Jobsub ID265273.0@dunegpschedd02.fnal.gov
Workflow ID11445
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-31 16:14:03
SiteCZ_FZU
EntryNova_CZ_FZU_osgce1
Last heartbeat2025-12-31 19:48:12
From worker nodeHostnamemilan19.farm.particle.cz
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-31 16:33:04
Input filesmonte-carlo-011445-000001
JobscriptExit code0
Real time3h (11671s)
CPU time3h (11117s = 95%)
Max RSS bytes4151353344 (3959 MiB)
Outputting started2025-12-31 19:47:35
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11445/1/001/reco_stage1.root
Finished2025-12-31 19:48:12
Saved logsjustin-logs:265273.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

asedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 101545 traces, 0 summary
Retagger: tagging trace set: gauss with 78729 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 78729 traces tagged "gauss"
FrameSaver: q=2.80085e+06 n=1425389 tag=gauss
wclsFrameSaver: saving 101545 traces tagged "wiener"
FrameSaver: q=2.97102e+06 n=1282394 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 17927 XUs and 11661 XVs -> 111 XUVs
C:0 T:1 25897 XUs and 16870 XVs -> 2158 XUVs
C:0 T:2 39186 XUs and 44235 XVs -> 1527 XUVs
C:0 T:3 24964 XUs and 23768 XVs -> 114 XUVs
C:0 T:4 13747 XUs and 14267 XVs -> 150 XUVs
C:0 T:5 51633 XUs and 41901 XVs -> 3669 XUVs
C:0 T:6 108323 XUs and 140589 XVs -> 7526 XUVs
C:0 T:7 44432 XUs and 48847 XVs -> 337 XUVs
C:0 T:8 6973 XUs and 9777 XVs -> 129 XUVs
C:0 T:9 66148 XUs and 86594 XVs -> 10528 XUVs
C:0 T:10 16602 XUs and 29586 XVs -> 3583 XUVs
C:0 T:11 188 XUs and 210 XVs -> 64 XUVs
29896 XUVs total
9464 collection wire objects
27454 potential space points
Neighbour search...
2933066 tests to find 1350010 neighbours
Iterating with no regularization...
Begin: 6.87224e+07
0 4.65995e+07
1 4.55556e+07
2 4.53847e+07
3 4.53271e+07
4 4.53021e+07
Now with regularization...
Begin: 3.67903e+07
0 3.67639e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411457679
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411457679
fLifetime: 30000 [us]
31-Dec-2025 20:47:34 CET  Closed output file "reco_stage1.root"
31-Dec-2025 20:47:34 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          249.531       319.158       425.767       298.549       58.1835        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00133987    0.00253891    0.00326864    0.00276178    0.000655046      10     
reco:ophit:OpHitFinder                             0.256335      0.315704       0.35318      0.322689      0.0307201       10     
reco:opflash:OpFlashFinder                         0.0972265     0.155748      0.182538      0.167719      0.0292365       10     
reco:opslicer:OpSlicer                             0.519301      0.755625      0.938428      0.763705      0.110751        10     
reco:caldata:DataPrepModule                         4.66318       6.24547       6.95192       6.35405      0.582017        10     
reco:wclsdatasp:WireCellToolkit                     34.4914       36.6359       37.9318       36.9036       1.13149        10     
reco:gaushit:GausHitFinder                         0.804618       1.15737       1.50344       1.16119      0.194701        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000437418   0.000726204   0.00116682    0.000719302   0.000189186      10     
reco:reco3d:SpacePointSolver                        8.16611       15.7022       23.0654       15.8556       5.40219        10     
reco:hitpdune:DisambigFromSpacePoints              0.984978       1.5215        2.30745       1.3419       0.422643        10     
reco:pandora:StandardPandora                        15.3218       25.8468       44.791        21.8613       9.29004        10     
reco:pandoraWriter:StandardPandora                 0.395179      0.567917      0.847769      0.558258      0.136474        10     
reco:pandoraTrack:LArPandoraTrackCreation           2.1404        2.93185       4.06573       2.73414      0.612563        10     
reco:pandoraShower:LArPandoraShowerCreation         1.61556       2.74683       4.41265       2.41899       0.94577        10     
reco:pandoracalo:Calorimetry                        1.22564       1.69067       2.35757       1.50996       0.38603        10     
reco:pandoracalonosce:Calorimetry                   1.0843        1.52464       2.15404       1.36038      0.357257        10     
reco:pandorapid:Chi2ParticleID                     0.0050607    0.00642425    0.00927127    0.00606037    0.00114903       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0446579     0.0561641      0.08178      0.0513849     0.0110019       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00474529     0.0056191    0.00661188    0.00564171    0.000546044      10     
reco:pandoraShowercalo:ShowerCalorimetry            1.64055       2.89501       4.85247       2.46525       1.11252        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.60843       2.84322       4.75425       2.4111        1.09823        10     
reco:emtrkmichelid:EmTrackMichelId                  149.161       183.442       238.614       172.738       29.9474        10     
reco:crttag:SingleCRTMatchingProducer             0.00772277     0.0120952     0.0253347    0.00990813    0.00515114       10     
reco:crtreco:TwoCRTMatchingProducer               0.00556133    0.00693021     0.0100988    0.00645231    0.00129792       10     
reco:anodepiercerst0:T0RecoAnodePiercers            0.61169      0.888029       1.24353      0.850112       0.16404        10     
reco:pandora2Track:LArPandoraTrackCreation          5.83043       8.5353        13.0588       7.83775       2.34237        10     
reco:pandora2calo:Calorimetry                       2.9157        4.42285       6.93925       3.8252        1.43739        10     
reco:pandora2calonosce:Calorimetry                  2.73017       4.20032       6.61176       3.63081       1.39291        10     
reco:pandora2pid:Chi2ParticleID                    0.011296      0.0140849     0.0179921     0.0139359    0.00202818       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0529319     0.0687621     0.0923568     0.0631074     0.0124003       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.011121      0.0136901     0.0175118     0.0134467    0.00195695       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.22371       3.27316       4.97515       2.94319      0.938481        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.54611       3.91108       6.11964       3.38628       1.2913         10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.36913       3.62481       5.70932       3.13041       1.21088        10     
[art]:TriggerResults:TriggerResultInserter         2.54e-05     4.96309e-05   0.00014254    4.14445e-05   3.21521e-05      10     
end_path:out1:RootOutput                            7.2e-06     9.4049e-06     1.887e-05     8.435e-06    3.2093e-06       10     
end_path:out1:RootOutput(write)                     2.35963       3.07592       3.93737       3.04121      0.490452        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4963.57 MB
  Peak resident set size usage (VmHWM): 4151.35 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.10 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.29 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.82 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9683224
-rw-r--r-- 1 fermilab users 2289098599 Dec 31 19:51 detsim_stage2.root
-rw-r--r-- 1 fermilab users 2175810096 Dec 31 19:48 detsim_stage1.root
-rw-r--r-- 1 fermilab users 1938403423 Dec 31 20:47 reco_stage1.root
-rw-r--r-- 1 fermilab users 1760249448 Dec 31 17:54 g4_stage2.root
-rw-r--r-- 1 fermilab users 1685862774 Dec 31 17:40 g4_stage1.root
-rw-r--r-- 1 fermilab users   65568490 Dec 31 20:44 Pandora_Events.pndr
-rw-r--r-- 1 fermilab users     242747 Dec 31 20:47 jobscript.log
-rw-r--r-- 1 fermilab users     217088 Dec 31 20:47 mem.db
-rw-r--r-- 1 fermilab users      62376 Dec 31 20:47 Pandora_Geometry.xml
-rw-r--r-- 1 fermilab users      36864 Dec 31 20:47 time.db
-rw-r--r-- 1 fermilab users        519 Dec 31 19:51 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 fermilab users        519 Dec 31 17:54 g4_protoDUNE_hist.root
-rw-r--r-- 1 fermilab users        519 Dec 31 20:47 protoDUNE_reco1_hist.root
-rw-r--r-- 1 fermilab users         52 Dec 31 17:33 all-input-dids.txt
-rw-r--r-- 1 fermilab users          7 Dec 31 20:47 justin-processed-pfns.txt
-rw-r--r-- 1 fermilab users          0 Dec 31 19:58 ProtoDUNE.log
-rw-r--r-- 1 fermilab users          0 Dec 31 19:51 debugprod.log
justIN time: 2026-02-06 23:31:51 UTC       justIN version: 01.06.00