justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 14069.1@dunegpschedd02.fnal.gov

Jobsub ID14069.1@dunegpschedd02.fnal.gov
Workflow ID254
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-08-02 23:05:48
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2025-08-03 03:29:39
From worker nodeHostnamewn-la-08.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-02 23:07:13
Input filesmonte-carlo-000254-000001
JobscriptExit code0
Real time4h (15638s)
CPU time4h (15416s = 98%)
Max RSS bytes4161622016 (3968 MiB)
Outputting started2025-08-03 03:27:52
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/00254/1/001/reco_stage1.root
Finished2025-08-03 03:29:39
Saved logsjustin-logs:14069.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

olBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 139422 traces, 0 summary
Retagger: tagging trace set: gauss with 108003 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 108003 traces tagged "gauss"
FrameSaver: q=3.56771e+06 n=1893254 tag=gauss
wclsFrameSaver: saving 139422 traces tagged "wiener"
FrameSaver: q=3.76716e+06 n=1672856 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 34570 XUs and 28792 XVs -> 305 XUVs
C:0 T:1 74367 XUs and 60366 XVs -> 4653 XUVs
C:0 T:2 98040 XUs and 133271 XVs -> 6621 XUVs
C:0 T:3 49919 XUs and 48047 XVs -> 490 XUVs
C:0 T:4 10638 XUs and 9666 XVs -> 84 XUVs
C:0 T:5 40514 XUs and 30714 XVs -> 4659 XUVs
C:0 T:6 142740 XUs and 141341 XVs -> 7146 XUVs
C:0 T:7 57417 XUs and 56313 XVs -> 652 XUVs
C:0 T:8 7330 XUs and 8328 XVs -> 62 XUVs
C:0 T:9 45357 XUs and 66031 XVs -> 3662 XUVs
C:0 T:10 37585 XUs and 22101 XVs -> 3024 XUVs
C:0 T:11 531 XUs and 369 XVs -> 54 XUVs
31412 XUVs total
12253 collection wire objects
28683 potential space points
Neighbour search...
2013223 tests to find 994300 neighbours
Iterating with no regularization...
Begin: 6.39155e+07
0 5.56676e+07
1 5.49936e+07
2 5.49148e+07
3 5.48953e+07
Now with regularization...
Begin: 4.48085e+07
0 4.48122e+07
Warning: metric increased
run: 1 ; subrun: 0 ; event: 10
evttime: 408425483
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 408425483
fLifetime: 30000 [us]
03-Aug-2025 05:27:50 CEST  Closed output file "reco_stage1.root"
03-Aug-2025 05:27:50 CEST  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          333.432       453.694       572.983       452.927       72.8613        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00108102     0.0051831     0.0081238    0.00538501     0.0021038       10     
reco:ophit:OpHitFinder                             0.341264      0.382916      0.418765      0.389895      0.0283833       10     
reco:opflash:OpFlashFinder                         0.175582      0.236117      0.293084       0.24697      0.0338213       10     
reco:opslicer:OpSlicer                             0.614327       0.76664      0.951625      0.773907      0.104894        10     
reco:caldata:DataPrepModule                         6.25127       6.41964       6.77672       6.3885       0.152175        10     
reco:wclsdatasp:WireCellToolkit                     43.1456       54.5752       61.4161       56.4379       6.53271        10     
reco:gaushit:GausHitFinder                          1.15462       1.31864       1.60477       1.33336      0.136206        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000405135   0.000544301   0.000767508   0.000538791   0.000108298      10     
reco:reco3d:SpacePointSolver                        12.2423       18.6737       27.3402       18.5989       5.07897        10     
reco:hitpdune:DisambigFromSpacePoints               1.21886       1.68959       2.44606       1.58657      0.390617        10     
reco:pandora:StandardPandora                        27.4632       49.4944       131.888       37.7787       29.9687        10     
reco:pandoraWriter:StandardPandora                 0.409568      0.489788      0.620607      0.484194      0.0603223       10     
reco:pandoraTrack:LArPandoraTrackCreation           3.75386       6.94188       9.25268       7.62855       1.68994        10     
reco:pandoraShower:LArPandoraShowerCreation         4.0182        8.14188       14.3876       8.17627       2.62789        10     
reco:pandoracalo:Calorimetry                        2.3614        3.69569       5.10973       3.68741      0.756436        10     
reco:pandoracalonosce:Calorimetry                   2.14002       3.44867       4.63163       3.52559      0.720241        10     
reco:pandorapid:Chi2ParticleID                    0.00561117     0.0063708    0.00748507    0.00625188    0.000513812      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0434189     0.053904      0.0688528     0.0523385    0.00675086       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00538479    0.00606425    0.00703389    0.00602543    0.000475395      10     
reco:pandoraShowercalo:ShowerCalorimetry            3.27414       7.86016       12.9411        7.918        2.61776        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       3.03132       7.4692        13.1393       7.39753       2.59732        10     
reco:emtrkmichelid:EmTrackMichelId                  182.014       214.443       260.936       210.244       23.176         10     
reco:crttag:SingleCRTMatchingProducer              0.0128669     0.0189866     0.0355468     0.016909     0.00635897       10     
reco:crtreco:TwoCRTMatchingProducer                0.0103783     0.0143104     0.030186      0.0121237    0.00554616       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.683654       0.99614       1.18332       1.03836      0.147733        10     
reco:pandora2Track:LArPandoraTrackCreation          11.016        20.5058       28.1486       20.153        4.91908        10     
reco:pandora2calo:Calorimetry                       4.93948       9.98693       14.4658       9.76844       2.62016        10     
reco:pandora2calonosce:Calorimetry                  4.61336       9.70192       13.898        9.58464       2.65979        10     
reco:pandora2pid:Chi2ParticleID                    0.0109709     0.0141094     0.0168193     0.0143845    0.00186515       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0598323     0.0693796     0.0863764     0.0679833     0.0080789       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0107568     0.0141793     0.0168908     0.0146801    0.00197546       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.1734        7.27874       11.3368       7.06614       1.84777        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.16455       8.21018       12.2343       7.79312       2.1448         10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.9372        7.17933       11.0213       6.77184       1.77561        10     
[art]:TriggerResults:TriggerResultInserter        3.5533e-05    5.62318e-05   0.000137304   4.5208e-05    2.83936e-05      10     
end_path:out1:RootOutput                           7.221e-06    8.6692e-06    1.4682e-05    8.0275e-06    2.06844e-06      10     
end_path:out1:RootOutput(write)                     2.92437       3.35889       4.12946       3.23616      0.388265        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4965.96 MB
  Peak resident set size usage (VmHWM): 4161.62 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.28 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.00 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10134692
-rw-r--r--. 1 dune003 dune 2378718106 Aug  3 04:10 detsim_stage2.root
-rw-r--r--. 1 dune003 dune 2264184046 Aug  3 04:07 detsim_stage1.root
-rw-r--r--. 1 dune003 dune 2059801649 Aug  3 05:27 reco_stage1.root
-rw-r--r--. 1 dune003 dune 1841134468 Aug  3 01:29 g4_stage2.root
-rw-r--r--. 1 dune003 dune 1759136402 Aug  3 01:14 g4_stage1.root
-rw-r--r--. 1 dune003 dune   74334940 Aug  3 05:22 Pandora_Events.pndr
-rw-r--r--. 1 dune003 dune     243185 Aug  3 05:27 jobscript.log
-rw-r--r--. 1 dune003 dune     217088 Aug  3 05:27 mem.db
-rw-r--r--. 1 dune003 dune      62376 Aug  3 05:27 Pandora_Geometry.xml
-rw-r--r--. 1 dune003 dune      36864 Aug  3 05:27 time.db
-rw-r--r--. 1 dune003 dune        519 Aug  3 04:10 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune003 dune        519 Aug  3 01:29 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune003 dune        519 Aug  3 05:27 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune003 dune         52 Aug  3 01:07 all-input-dids.txt
-rw-r--r--. 1 dune003 dune          7 Aug  3 05:27 justin-processed-pfns.txt
-rw-r--r--. 1 dune003 dune          0 Aug  3 04:15 ProtoDUNE.log
-rw-r--r--. 1 dune003 dune          0 Aug  3 04:10 debugprod.log
justIN time: 2025-08-04 16:02:47 UTC       justIN version: 01.04.00