justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 241413.0@dunegpschedd02.fnal.gov

Jobsub ID241413.0@dunegpschedd02.fnal.gov
Workflow ID9943
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-11-11 18:26:26
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2025-11-11 21:32:32
From worker nodeHostnamecomp15-01
cpuinfoAMD EPYC 9654 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-11 18:28:08
Input filesmonte-carlo-009943-000001
JobscriptExit code0
Real time2h (10761s)
CPU time2h (10317s = 95%)
Max RSS bytes4227809280 (4031 MiB)
Outputting started2025-11-11 21:27:31
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09943/1/001/reco_stage1.root
Finished2025-11-11 21:32:32
Saved logsjustin-logs:241413.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

lBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 171212 traces, 0 summary
Retagger: tagging trace set: gauss with 129932 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 129932 traces tagged "gauss"
FrameSaver: q=4.38485e+06 n=2306424 tag=gauss
wclsFrameSaver: saving 171212 traces tagged "wiener"
FrameSaver: q=4.63693e+06 n=2031034 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 50669 XUs and 56647 XVs -> 1035 XUVs
C:0 T:1 120977 XUs and 121849 XVs -> 12481 XUVs
C:0 T:2 106310 XUs and 189744 XVs -> 10694 XUVs
C:0 T:3 50548 XUs and 57745 XVs -> 400 XUVs
C:0 T:4 13605 XUs and 13082 XVs -> 199 XUVs
C:0 T:5 79678 XUs and 103553 XVs -> 14417 XUVs
C:0 T:6 158851 XUs and 122458 XVs -> 9973 XUVs
C:0 T:7 42894 XUs and 39694 XVs -> 477 XUVs
C:0 T:8 13371 XUs and 9764 XVs -> 172 XUVs
C:0 T:9 96725 XUs and 101682 XVs -> 7270 XUVs
C:0 T:10 83878 XUs and 63803 XVs -> 7716 XUVs
C:0 T:11 217 XUs and 192 XVs -> 23 XUVs
64857 XUVs total
16569 collection wire objects
59757 potential space points
Neighbour search...
9598493 tests to find 4327790 neighbours
Iterating with no regularization...
Begin: 9.03983e+07
0 7.09667e+07
1 6.98848e+07
2 6.97437e+07
3 6.96968e+07
Now with regularization...
Begin: 5.83293e+07
0 5.82804e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410452204
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410452204
fLifetime: 30000 [us]
11-Nov-2025 21:25:06 GMT  Closed output file "reco_stage1.root"
11-Nov-2025 21:25:06 GMT  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          208.464       300.331       470.619       262.019       85.0836        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00135731    0.00346633     0.0047211    0.00376123    0.00103634       10     
reco:ophit:OpHitFinder                             0.227548      0.290775      0.394795      0.277156      0.0467409       10     
reco:opflash:OpFlashFinder                          0.10147      0.167791      0.281339      0.159605      0.0472211       10     
reco:opslicer:OpSlicer                              0.43109      0.605638      0.873359      0.583358      0.125367        10     
reco:caldata:DataPrepModule                         4.69241       4.96367       5.40219       4.89929      0.244288        10     
reco:wclsdatasp:WireCellToolkit                     30.1337       31.6837       33.0104       31.6727      0.728275        10     
reco:gaushit:GausHitFinder                         0.747591      0.998787       1.2815       0.935088      0.181739        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000264372   0.000343659   0.000546355   0.000310303   8.32894e-05      10     
reco:reco3d:SpacePointSolver                        7.92342       13.1394       21.9512       11.1892       4.73195        10     
reco:hitpdune:DisambigFromSpacePoints              0.709474       1.28436       2.20369       1.08623      0.512932        10     
reco:pandora:StandardPandora                        15.1203       27.6043       58.2173       20.3509       14.0424        10     
reco:pandoraWriter:StandardPandora                 0.249822       6.10138       58.0663      0.306447       17.3218        10     
reco:pandoraTrack:LArPandoraTrackCreation           1.6757        2.64653       4.32059       2.35378      0.817835        10     
reco:pandoraShower:LArPandoraShowerCreation         1.46345       3.19406       6.30705       2.39709       1.53271        10     
reco:pandoracalo:Calorimetry                        0.99082       1.61581       2.55229       1.49754      0.508166        10     
reco:pandoracalonosce:Calorimetry                  0.903871       1.49681       2.35125       1.39487      0.472865        10     
reco:pandorapid:Chi2ParticleID                    0.00357676    0.00430847    0.00598996    0.00403727    0.000748477      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0339466     0.0444783     0.0587084     0.0419194    0.00823432       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00345845    0.00419893    0.00551923    0.00407333    0.000656337      10     
reco:pandoraShowercalo:ShowerCalorimetry            1.46648       3.52301       6.80897       2.69541       1.75993        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.42858       3.44156        6.716        2.63321       1.73158        10     
reco:emtrkmichelid:EmTrackMichelId                  120.639       163.882       226.17        151.702       35.0924        10     
reco:crttag:SingleCRTMatchingProducer             0.00773818     0.0116848     0.0190972     0.010791     0.00314157       10     
reco:crtreco:TwoCRTMatchingProducer               0.00472909    0.00658025    0.00925875    0.00613617    0.00146798       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.579313      0.763327       0.95667      0.709477      0.150499        10     
reco:pandora2Track:LArPandoraTrackCreation          4.58848       9.02605       15.9772       7.07759       3.81583        10     
reco:pandora2calo:Calorimetry                       2.31311       4.8644        8.57597       3.99119       2.09864        10     
reco:pandora2calonosce:Calorimetry                  2.18604       4.66711       8.25044       3.80417       2.04425        10     
reco:pandora2pid:Chi2ParticleID                   0.00768518     0.0102931     0.0139344    0.00959971    0.00216603       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0413985     0.0554835     0.0741205     0.0498579     0.0110501       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00742021    0.00988795     0.0133434    0.00899859    0.00206109       10     
reco:pandora2Shower:LArPandoraShowerCreation        1.7905        3.39951       5.89856       2.68866       1.32868        10     
reco:pandora2Showercalo:ShowerCalorimetry           2.02554       4.23859       7.40159       3.50083       1.79669        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      1.86919       3.93819       6.94403       3.21785       1.67259        10     
[art]:TriggerResults:TriggerResultInserter         4.076e-05    5.22184e-05   0.000101041   4.5705e-05    1.74672e-05      10     
end_path:out1:RootOutput                           1.076e-05    1.25863e-05    1.637e-05    1.1945e-05    1.67218e-06      10     
end_path:out1:RootOutput(write)                     1.88622       2.56314       3.61356       2.35586      0.565609        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5039.62 MB
  Peak resident set size usage (VmHWM): 4227.81 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.21 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   2.97 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.12 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9542231
-rw-r--r-- 1 pltdune004 pltdune 2268003831 Nov 11 20:33 detsim_stage2.root
-rw-r--r-- 1 pltdune004 pltdune 2155474145 Nov 11 20:31 detsim_stage1.root
-rw-r--r-- 1 pltdune004 pltdune 1949073344 Nov 11 21:25 reco_stage1.root
-rw-r--r-- 1 pltdune004 pltdune 1697670267 Nov 11 18:43 g4_stage2.root
-rw-r--r-- 1 pltdune004 pltdune 1626729440 Nov 11 18:33 g4_stage1.root
-rw-r--r-- 1 pltdune004 pltdune   73728070 Nov 11 21:19 Pandora_Events.pndr
-rw-r--r-- 1 pltdune004 pltdune     242756 Nov 11 21:27 jobscript.log
-rw-r--r-- 1 pltdune004 pltdune     217088 Nov 11 21:27 mem.db
-rw-r--r-- 1 pltdune004 pltdune      62376 Nov 11 21:27 Pandora_Geometry.xml
-rw-r--r-- 1 pltdune004 pltdune      36864 Nov 11 21:27 time.db
-rw-r--r-- 1 pltdune004 pltdune        519 Nov 11 20:33 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 pltdune004 pltdune        519 Nov 11 18:43 g4_protoDUNE_hist.root
-rw-r--r-- 1 pltdune004 pltdune        519 Nov 11 21:27 protoDUNE_reco1_hist.root
-rw-r--r-- 1 pltdune004 pltdune         52 Nov 11 18:28 all-input-dids.txt
-rw-r--r-- 1 pltdune004 pltdune          7 Nov 11 21:27 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune004 pltdune          0 Nov 11 20:41 ProtoDUNE.log
-rw-r--r-- 1 pltdune004 pltdune          0 Nov 11 20:33 debugprod.log
justIN time: 2026-02-10 07:57:26 UTC       justIN version: 01.06.00