justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 249058.0@dunegpschedd02.fnal.gov

Jobsub ID249058.0@dunegpschedd02.fnal.gov
Workflow ID10443
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-21 00:17:44
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc01
Last heartbeat2025-11-21 04:06:10
From worker nodeHostnamewn-lb-08.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-21 00:19:10
Input filesmonte-carlo-010443-000001
JobscriptExit code0
Real time3h (13538s)
CPU time3h (13411s = 99%)
Max RSS bytes3924230144 (3742 MiB)
Outputting started2025-11-21 04:04:48
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/10443/1/001/reco_stage1.root
Finished2025-11-21 04:06:10
Saved logsjustin-logs:249058.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

cKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 150719 traces, 0 summary
Retagger: tagging trace set: gauss with 111134 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 111134 traces tagged "gauss"
FrameSaver: q=4.04377e+06 n=1968672 tag=gauss
wclsFrameSaver: saving 150719 traces tagged "wiener"
FrameSaver: q=4.31888e+06 n=1747947 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 49498 XUs and 45630 XVs -> 568 XUVs
C:0 T:1 78248 XUs and 79320 XVs -> 6226 XUVs
C:0 T:2 114152 XUs and 130358 XVs -> 4571 XUVs
C:0 T:3 57469 XUs and 60593 XVs -> 424 XUVs
C:0 T:4 12164 XUs and 10358 XVs -> 166 XUVs
C:0 T:5 50234 XUs and 32935 XVs -> 9027 XUVs
C:0 T:6 60525 XUs and 88529 XVs -> 5561 XUVs
C:0 T:7 21989 XUs and 30341 XVs -> 317 XUVs
C:0 T:8 16839 XUs and 17490 XVs -> 824 XUVs
C:0 T:9 165746 XUs and 161079 XVs -> 20716 XUVs
C:0 T:10 34184 XUs and 26732 XVs -> 6087 XUVs
C:0 T:11 140 XUs and 110 XVs -> 12 XUVs
54499 XUVs total
12719 collection wire objects
50043 potential space points
Neighbour search...
6559181 tests to find 3003726 neighbours
Iterating with no regularization...
Begin: 8.0668e+07
0 6.80332e+07
1 6.70317e+07
2 6.6882e+07
3 6.68379e+07
Now with regularization...
Begin: 5.49556e+07
0 5.49067e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410639634
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410639634
fLifetime: 30000 [us]
21-Nov-2025 05:04:46 CET  Closed output file "reco_stage1.root"
21-Nov-2025 05:04:46 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          269.87        387.755       540.769       379.644       80.9622        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00111243     0.0047419    0.00813766    0.00497955    0.00223487       10     
reco:ophit:OpHitFinder                             0.326071      0.357802      0.400299      0.363418      0.0232221       10     
reco:opflash:OpFlashFinder                         0.167694      0.210874      0.259346       0.20559      0.0298849       10     
reco:opslicer:OpSlicer                             0.505447      0.692302      0.870529      0.702623      0.0967799       10     
reco:caldata:DataPrepModule                         6.20753       6.40032       6.79572       6.3814        0.16246        10     
reco:wclsdatasp:WireCellToolkit                     41.6979       48.8667       52.0418       50.0608       3.62987        10     
reco:gaushit:GausHitFinder                         0.991578       1.22959       1.63738       1.20073       0.19236        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000381873   0.000510941   0.000826468   0.000465311   0.000136113      10     
reco:reco3d:SpacePointSolver                        8.49899       23.0516       41.5772       19.5926       11.082         10     
reco:hitpdune:DisambigFromSpacePoints              0.835889       1.49044       2.41142       1.43049      0.451058        10     
reco:pandora:StandardPandora                        15.9139       26.8568       41.3887       23.904        8.79053        10     
reco:pandoraWriter:StandardPandora                 0.333184      0.444839      0.613137      0.433788      0.078662        10     
reco:pandoraTrack:LArPandoraTrackCreation           2.26541       4.99984       7.58346       4.54847       1.76676        10     
reco:pandoraShower:LArPandoraShowerCreation         2.5441        5.61237       9.26587       4.85289       2.39026        10     
reco:pandoracalo:Calorimetry                        1.16769       2.72536       4.5063        2.38714       1.04132        10     
reco:pandoracalonosce:Calorimetry                   1.01395       2.48706       4.2304        2.29897      0.984588        10     
reco:pandorapid:Chi2ParticleID                     0.0039442    0.00561094    0.00656568    0.00581049    0.00071131       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0407035     0.0510934     0.0642607     0.0493754    0.00784066       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00372425    0.00533004    0.00605037    0.00545647    0.000671511      10     
reco:pandoraShowercalo:ShowerCalorimetry            2.16569       5.68401       10.0971       5.09852       2.83911        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.16591       5.38239       9.17616       4.80877       2.60799        10     
reco:emtrkmichelid:EmTrackMichelId                  146.997       197.145       259.945       196.203       32.2859        10     
reco:crttag:SingleCRTMatchingProducer              0.0116523     0.0192686     0.0336485     0.0168493    0.00663013       10     
reco:crtreco:TwoCRTMatchingProducer               0.00803528     0.0134645     0.0276016     0.0110451    0.00570785       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.735043      0.951829       1.18582      0.934209      0.128914        10     
reco:pandora2Track:LArPandoraTrackCreation          7.15464       16.9196       31.4924       16.4193       7.40627        10     
reco:pandora2calo:Calorimetry                       3.1877        7.60059       15.0269       7.44768       3.56033        10     
reco:pandora2calonosce:Calorimetry                  2.94212       6.95929       14.2308       6.5995         3.372         10     
reco:pandora2pid:Chi2ParticleID                   0.00884007     0.012086      0.0145983     0.0120418    0.00161725       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0496303     0.0631867     0.0814719     0.0605515     0.0104028       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00871723     0.0122073     0.0146435     0.0123447     0.0016843       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.63278       6.00342       11.7809       5.20794       2.77023        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.73512       6.42489       13.2219       6.02639       3.07026        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.56199       5.74229       12.3475       5.4517        2.79656        10     
[art]:TriggerResults:TriggerResultInserter        4.2243e-05    5.50483e-05   0.000126589   4.7857e-05    2.40124e-05      10     
end_path:out1:RootOutput                           8.092e-06    9.5092e-06     1.399e-05    8.9185e-06    1.64456e-06      10     
end_path:out1:RootOutput(write)                     2.33381       3.12549       4.04163       3.01214      0.522469        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4723.77 MB
  Peak resident set size usage (VmHWM): 3924.23 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.28 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.98 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 8928140
-rw-r--r--. 1 dune009 dune 2147408498 Nov 21 03:58 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2040511368 Nov 21 03:56 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 1804241258 Nov 21 05:04 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1580815465 Nov 21 01:37 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1501793622 Nov 21 01:25 g4_stage1.root
-rw-r--r--. 1 dune009 dune   67032590 Nov 21 04:58 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242985 Nov 21 05:04 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Nov 21 05:04 mem.db
-rw-r--r--. 1 dune009 dune      62376 Nov 21 05:04 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Nov 21 05:04 time.db
-rw-r--r--. 1 dune009 dune        519 Nov 21 03:58 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov 21 01:37 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Nov 21 05:04 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Nov 21 01:19 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Nov 21 05:04 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Nov 21 04:03 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Nov 21 03:58 debugprod.log
justIN time: 2026-02-09 01:58:13 UTC       justIN version: 01.06.00