justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 265241.0@dunegpschedd02.fnal.gov

Jobsub ID265241.0@dunegpschedd02.fnal.gov
Workflow ID11439
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-31 08:49:39
SiteCZ_FZU
EntryNova_CZ_FZU_osgce2
Last heartbeat2025-12-31 12:10:55
From worker nodeHostnamemilan04.farm.particle.cz
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-31 08:50:06
Input filesmonte-carlo-011439-000001
JobscriptExit code0
Real time3h (11740s)
CPU time3h (11235s = 95%)
Max RSS bytes4123963392 (3932 MiB)
Outputting started2025-12-31 12:05:48
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11439/1/001/reco_stage1.root
Finished2025-12-31 12:10:55
Saved logsjustin-logs:265241.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

gnalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 111776 traces, 0 summary
Retagger: tagging trace set: gauss with 86042 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 86042 traces tagged "gauss"
FrameSaver: q=3.03968e+06 n=1520603 tag=gauss
wclsFrameSaver: saving 111776 traces tagged "wiener"
FrameSaver: q=3.20959e+06 n=1347048 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 40088 XUs and 31999 XVs -> 399 XUVs
C:0 T:1 100521 XUs and 80277 XVs -> 7488 XUVs
C:0 T:2 121495 XUs and 127131 XVs -> 6964 XUVs
C:0 T:3 60203 XUs and 53244 XVs -> 511 XUVs
C:0 T:4 6854 XUs and 8213 XVs -> 117 XUVs
C:0 T:5 36255 XUs and 45080 XVs -> 10543 XUVs
C:0 T:6 89222 XUs and 104403 XVs -> 3671 XUVs
C:0 T:7 34181 XUs and 39725 XVs -> 274 XUVs
C:0 T:8 4365 XUs and 4136 XVs -> 104 XUVs
C:0 T:9 23687 XUs and 29351 XVs -> 2638 XUVs
C:0 T:10 21828 XUs and 12047 XVs -> 2461 XUVs
C:0 T:11 118 XUs and 69 XVs -> 12 XUVs
35182 XUVs total
9400 collection wire objects
32310 potential space points
Neighbour search...
4690528 tests to find 2149464 neighbours
Iterating with no regularization...
Begin: 6.93911e+07
0 5.86699e+07
1 5.79687e+07
2 5.78659e+07
3 5.7836e+07
Now with regularization...
Begin: 4.90475e+07
0 4.90159e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411451423
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411451423
fLifetime: 30000 [us]
31-Dec-2025 13:05:46 CET  Closed output file "reco_stage1.root"
31-Dec-2025 13:05:46 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          292.279       359.691       435.357       356.276       42.9372        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.0010642    0.00309032     0.0045711    0.00373359    0.00122446       10     
reco:ophit:OpHitFinder                             0.213057      0.318622      0.377284      0.329045      0.0458948       10     
reco:opflash:OpFlashFinder                         0.110518      0.161002      0.212981      0.152427      0.0332988       10     
reco:opslicer:OpSlicer                             0.416168      0.755016      0.986241      0.789243      0.143598        10     
reco:caldata:DataPrepModule                         5.63605       6.2891        6.61187       6.32043      0.255699        10     
reco:wclsdatasp:WireCellToolkit                     33.5555       34.968        36.8933       34.7653       0.9813         10     
reco:gaushit:GausHitFinder                          1.03792       1.30333       1.66483       1.25158      0.195591        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000364999   0.000543252   0.000831927   0.000523284   0.000123683      10     
reco:reco3d:SpacePointSolver                        10.1658       16.1441       28.9921       15.4268       5.29823        10     
reco:hitpdune:DisambigFromSpacePoints               1.05577       1.83381       2.66983       1.79048      0.487815        10     
reco:pandora:StandardPandora                        23.1545       35.8166       54.1867       33.8572       9.72823        10     
reco:pandoraWriter:StandardPandora                 0.373315      0.534296      0.738672      0.483151      0.123963        10     
reco:pandoraTrack:LArPandoraTrackCreation           2.52916       3.20131       4.31606       3.15854      0.447782        10     
reco:pandoraShower:LArPandoraShowerCreation         2.58826       3.50703       5.53004       3.33594      0.817769        10     
reco:pandoracalo:Calorimetry                        1.47431       1.93037       2.55704       1.89615      0.288179        10     
reco:pandoracalonosce:Calorimetry                   1.33558       1.73687       2.29002       1.73988      0.253539        10     
reco:pandorapid:Chi2ParticleID                    0.00530236    0.00615638    0.00745176    0.00605898    0.00061542       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0496544     0.065623      0.0761241     0.0669594    0.00797681       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00527448    0.00587679    0.00687431    0.00571989    0.000553711      10     
reco:pandoraShowercalo:ShowerCalorimetry            2.49001       3.80091       6.01954       3.68298      0.993413        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.40323       3.72376       5.74707       3.60623      0.926492        10     
reco:emtrkmichelid:EmTrackMichelId                  171.435       204.551        236.3        206.106       20.8752        10     
reco:crttag:SingleCRTMatchingProducer             0.00932687     0.0127149     0.017094      0.0125089    0.00212433       10     
reco:crtreco:TwoCRTMatchingProducer               0.00584049    0.00752952    0.00919357    0.00771521    0.00104248       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.623463      0.879991       1.00593      0.927256      0.124989        10     
reco:pandora2Track:LArPandoraTrackCreation          8.07674       10.6691       15.2622       10.3793       2.01854        10     
reco:pandora2calo:Calorimetry                       3.86202       5.43685       7.89381       5.39538       1.08763        10     
reco:pandora2calonosce:Calorimetry                  3.76535       5.20242       7.6128        5.01012       1.0861         10     
reco:pandora2pid:Chi2ParticleID                    0.0133999     0.0150081     0.0181783     0.0147418    0.00129893       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0638286     0.079752      0.0985191     0.0785615    0.00982359       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0131858     0.014605      0.0173708     0.0141684    0.00111074       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.18606       4.00674       5.67642       3.88852      0.702912        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.48024       4.87283       7.06207       4.80604       1.00764        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.22616       4.51425       6.55027       4.47007      0.952833        10     
[art]:TriggerResults:TriggerResultInserter         4.158e-05    6.30679e-05   0.000134589   4.9685e-05    2.84744e-05      10     
end_path:out1:RootOutput                           8.71e-06      1.024e-05     1.462e-05     9.375e-06    1.98858e-06      10     
end_path:out1:RootOutput(write)                     2.5964        3.25361       3.93087       3.22317      0.438297        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4940.26 MB
  Peak resident set size usage (VmHWM): 4123.96 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.29 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.00 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9789724
-rw-r--r-- 1 fermilab users 2315086739 Dec 31 12:02 detsim_stage2.root
-rw-r--r-- 1 fermilab users 2197647513 Dec 31 12:00 detsim_stage1.root
-rw-r--r-- 1 fermilab users 2003845756 Dec 31 13:05 reco_stage1.root
-rw-r--r-- 1 fermilab users 1749974306 Dec 31 10:10 g4_stage2.root
-rw-r--r-- 1 fermilab users 1682010927 Dec 31 09:56 g4_stage1.root
-rw-r--r-- 1 fermilab users   75479930 Dec 31 13:02 Pandora_Events.pndr
-rw-r--r-- 1 fermilab users     242582 Dec 31 13:05 jobscript.log
-rw-r--r-- 1 fermilab users     217088 Dec 31 13:05 mem.db
-rw-r--r-- 1 fermilab users      62376 Dec 31 13:05 Pandora_Geometry.xml
-rw-r--r-- 1 fermilab users      36864 Dec 31 13:05 time.db
-rw-r--r-- 1 fermilab users        519 Dec 31 12:02 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 fermilab users        519 Dec 31 10:10 g4_protoDUNE_hist.root
-rw-r--r-- 1 fermilab users        519 Dec 31 13:05 protoDUNE_reco1_hist.root
-rw-r--r-- 1 fermilab users         52 Dec 31 09:50 all-input-dids.txt
-rw-r--r-- 1 fermilab users          7 Dec 31 13:05 justin-processed-pfns.txt
-rw-r--r-- 1 fermilab users          0 Dec 31 12:10 ProtoDUNE.log
-rw-r--r-- 1 fermilab users          0 Dec 31 12:02 debugprod.log
justIN time: 2026-02-06 23:31:52 UTC       justIN version: 01.06.00