justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 273625.0@dunegpschedd01.fnal.gov

Jobsub ID273625.0@dunegpschedd01.fnal.gov
Workflow ID11687
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-10 17:45:48
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2026-01-10 22:39:54
From worker nodeHostnamehnode77.pic.es
cpuinfoAMD EPYC 7443P 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-10 17:46:25
Input filesmonte-carlo-011687-000001
JobscriptExit code0
Real time4h (17398s)
CPU time3h (11203s = 64%)
Max RSS bytes4488794112 (4280 MiB)
Outputting started2026-01-10 22:36:24
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11687/1/001/reco_stage1.root
Finished2026-01-10 22:39:54
Saved logsjustin-logs:273625.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 158942 traces, 0 summary
Retagger: tagging trace set: gauss with 118033 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 118033 traces tagged "gauss"
FrameSaver: q=4.28292e+06 n=2035589 tag=gauss
wclsFrameSaver: saving 158942 traces tagged "wiener"
FrameSaver: q=4.5818e+06 n=1787735 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 50539 XUs and 47616 XVs -> 912 XUVs
C:0 T:1 136599 XUs and 112574 XVs -> 19532 XUVs
C:0 T:2 292929 XUs and 255477 XVs -> 32446 XUVs
C:0 T:3 95617 XUs and 79332 XVs -> 1200 XUVs
C:0 T:4 10475 XUs and 11985 XVs -> 227 XUVs
C:0 T:5 91956 XUs and 61720 XVs -> 21507 XUVs
C:0 T:6 103283 XUs and 104061 XVs -> 5967 XUVs
C:0 T:7 42654 XUs and 44304 XVs -> 354 XUVs
C:0 T:8 4915 XUs and 5537 XVs -> 46 XUVs
C:0 T:9 59140 XUs and 61330 XVs -> 11994 XUVs
C:0 T:10 30030 XUs and 27663 XVs -> 4697 XUVs
C:0 T:11 332 XUs and 284 XVs -> 56 XUVs
98938 XUVs total
14950 collection wire objects
86918 potential space points
Neighbour search...
21880330 tests to find 8842670 neighbours
Iterating with no regularization...
Begin: 1.01554e+08
0 7.11942e+07
1 6.81972e+07
2 6.76188e+07
3 6.74086e+07
4 6.73025e+07
5 6.7239e+07
Now with regularization...
Begin: 5.23847e+07
0 5.21956e+07
1 5.2075e+07
2 5.19857e+07
3 5.19146e+07
4 5.1857e+07
5 5.18099e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411660158
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411660158
fLifetime: 30000 [us]
10-Jan-2026 23:36:22 CET  Closed output file "reco_stage1.root"
10-Jan-2026 23:36:22 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          172.853       466.237       990.893       409.111       220.72         10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00118561    0.00317598     0.0057103    0.00313018    0.00130768       10     
reco:ophit:OpHitFinder                             0.174583      0.303737      0.396183      0.305598      0.0630852       10     
reco:opflash:OpFlashFinder                         0.101777      0.224151      0.279774      0.224687      0.0473198       10     
reco:opslicer:OpSlicer                              0.33917      0.707724      0.988356      0.739808      0.163846        10     
reco:caldata:DataPrepModule                         3.79684       4.96415       7.19066       4.5882        1.04448        10     
reco:wclsdatasp:WireCellToolkit                     24.3695       31.1129       45.8585       27.8394       7.00823        10     
reco:gaushit:GausHitFinder                          0.52729       1.22785       2.33433       1.18947      0.451006        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000233192   0.000505377   0.000790438   0.000506127   0.000154412      10     
reco:reco3d:SpacePointSolver                        5.10399        32.38        148.456       18.464        39.4657        10     
reco:hitpdune:DisambigFromSpacePoints              0.517873       1.74665       2.70531       1.72046      0.563949        10     
reco:pandora:StandardPandora                        9.88198       78.2872       292.523       42.2644       83.7977        10     
reco:pandoraWriter:StandardPandora                  0.1885       0.387637      0.670543       0.37615       0.11859        10     
reco:pandoraTrack:LArPandoraTrackCreation           1.62449       3.99864       5.89024       4.00625       1.19109        10     
reco:pandoraShower:LArPandoraShowerCreation         1.3255        4.45039       7.83706       3.61346       1.8365         10     
reco:pandoracalo:Calorimetry                        0.88885       2.36583       4.28122       2.19494      0.900047        10     
reco:pandoracalonosce:Calorimetry                  0.613574       2.20074       3.61638       2.12433      0.766154        10     
reco:pandorapid:Chi2ParticleID                    0.00280642    0.00615103    0.00860371    0.00644483    0.00190071       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0196579     0.0581228     0.0799379     0.0672644     0.0196189       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00259965    0.00602799    0.00887426    0.00627784     0.001933        10     
reco:pandoraShowercalo:ShowerCalorimetry            1.1527        17.2017       129.448       5.37408       37.4463        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.14604        4.802        8.83668       4.69087       1.93356        10     
reco:emtrkmichelid:EmTrackMichelId                   105.6        233.403       697.335       184.868       160.105        10     
reco:crttag:SingleCRTMatchingProducer             0.00404405     0.0132086     0.0253849     0.0116621    0.00616552       10     
reco:crtreco:TwoCRTMatchingProducer               0.00324589    0.00884317     0.0156778    0.00760054    0.00384411       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.472628       1.05977       1.48373       1.01656      0.292007        10     
reco:pandora2Track:LArPandoraTrackCreation          3.94898       13.1573       22.6269       12.3893       4.89535        10     
reco:pandora2calo:Calorimetry                       1.75189       7.00682       12.1405       6.68408       2.63356        10     
reco:pandora2calonosce:Calorimetry                  1.74052       6.52403       10.452        6.04759       2.24083        10     
reco:pandora2pid:Chi2ParticleID                   0.00928441     0.0134368     0.0200756     0.0124075    0.00364067       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0338326     0.0666926     0.0986217     0.0665276     0.0209717       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00704721     0.0137383     0.0199101     0.0139443    0.00391675       10     
reco:pandora2Shower:LArPandoraShowerCreation        1.73593       4.57791       7.13624       4.32519       1.52724        10     
reco:pandora2Showercalo:ShowerCalorimetry           1.63028       5.51104       8.54712       5.72983       1.7191         10     
reco:pandora2Showercalonosce:ShowerCalorimetry      1.56708       5.52962       7.68928       5.66144       1.65889        10     
[art]:TriggerResults:TriggerResultInserter         2.152e-05    3.16704e-05   8.4321e-05    2.44655e-05   1.80344e-05      10     
end_path:out1:RootOutput                           3.95e-06     4.7352e-06     8.181e-06    4.3105e-06    1.21189e-06      10     
end_path:out1:RootOutput(write)                     1.3536        2.80973       3.73085       2.91337      0.621194        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5309.77 MB
  Peak resident set size usage (VmHWM): 4488.79 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.21 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.17 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.12 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11229468
-rw-r--r-- 1 dune001 dune 2592820861 Jan 10 22:12 detsim_stage2.root
-rw-r--r-- 1 dune001 dune 2454370609 Jan 10 21:57 detsim_stage1.root
-rw-r--r-- 1 dune001 dune 2322158061 Jan 10 23:36 reco_stage1.root
-rw-r--r-- 1 dune001 dune 2069171290 Jan 10 19:32 g4_stage2.root
-rw-r--r-- 1 dune001 dune 1974441052 Jan 10 18:58 g4_stage1.root
-rw-r--r-- 1 dune001 dune   85383890 Jan 10 23:33 Pandora_Events.pndr
-rw-r--r-- 1 dune001 dune     242963 Jan 10 23:36 jobscript.log
-rw-r--r-- 1 dune001 dune     217088 Jan 10 23:36 mem.db
-rw-r--r-- 1 dune001 dune      62376 Jan 10 23:36 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 dune      36864 Jan 10 23:36 time.db
-rw-r--r-- 1 dune001 dune        519 Jan 10 22:12 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune        519 Jan 10 19:32 g4_protoDUNE_hist.root
-rw-r--r-- 1 dune001 dune        519 Jan 10 23:36 protoDUNE_reco1_hist.root
-rw-r--r-- 1 dune001 dune         52 Jan 10 18:46 all-input-dids.txt
-rw-r--r-- 1 dune001 dune          7 Jan 10 23:36 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune          0 Jan 10 22:21 ProtoDUNE.log
-rw-r--r-- 1 dune001 dune          0 Jan 10 22:12 debugprod.log
justIN time: 2026-02-06 11:41:51 UTC       justIN version: 01.06.00