justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 288560.0@dunegpschedd02.fnal.gov

Jobsub ID288560.0@dunegpschedd02.fnal.gov
Workflow ID12113
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-20 14:48:55
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2026-01-20 20:48:18
From worker nodeHostnamewn-lb-13.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-20 14:52:52
Input filesmonte-carlo-012113-000001
JobscriptExit code0
Real time5h (21215s)
CPU time5h (21020s = 99%)
Max RSS bytes5311848448 (5065 MiB)
Outputting started2026-01-20 20:46:28
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12113/1/001/reco_stage1.root
Finished2026-01-20 20:48:18
Saved logsjustin-logs:288560.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

cScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 157034 traces, 0 summary
Retagger: tagging trace set: gauss with 121558 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 121558 traces tagged "gauss"
FrameSaver: q=4.28365e+06 n=2169365 tag=gauss
wclsFrameSaver: saving 157034 traces tagged "wiener"
FrameSaver: q=4.5038e+06 n=1919191 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 42194 XUs and 39179 XVs -> 419 XUVs
C:0 T:1 80148 XUs and 80747 XVs -> 7043 XUVs
C:0 T:2 98617 XUs and 115047 XVs -> 3689 XUVs
C:0 T:3 49623 XUs and 51940 XVs -> 319 XUVs
C:0 T:4 11572 XUs and 13420 XVs -> 90 XUVs
C:0 T:5 46960 XUs and 41507 XVs -> 3885 XUVs
C:0 T:6 143477 XUs and 110076 XVs -> 5320 XUVs
C:0 T:7 59734 XUs and 43279 XVs -> 511 XUVs
C:0 T:8 8806 XUs and 10454 XVs -> 155 XUVs
C:0 T:9 59920 XUs and 97908 XVs -> 7412 XUVs
C:0 T:10 82735 XUs and 37462 XVs -> 7875 XUVs
C:0 T:11 290 XUs and 181 XVs -> 26 XUVs
36744 XUVs total
14186 collection wire objects
34420 potential space points
Neighbour search...
3036898 tests to find 1405738 neighbours
Iterating with no regularization...
Begin: 9.91854e+07
0 8.37992e+07
1 8.27756e+07
2 8.26254e+07
3 8.25764e+07
Now with regularization...
Begin: 6.96687e+07
0 6.96447e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
run: 1 ; subrun: 0 ; event: 10
evttime: 411853287
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411853287
fLifetime: 30000 [us]
20-Jan-2026 21:46:24 CET  Closed output file "reco_stage1.root"
20-Jan-2026 21:46:24 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          528.627       731.282       1129.54       698.504       183.458        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00136316    0.00707854     0.0131732    0.00772157    0.00362838       10     
reco:ophit:OpHitFinder                              0.32223      0.414117      0.541034      0.396118      0.0627993       10     
reco:opflash:OpFlashFinder                         0.235591       0.35794      0.511117      0.337584      0.0813498       10     
reco:opslicer:OpSlicer                             0.678655      0.843281       1.1792        0.8293       0.135734        10     
reco:caldata:DataPrepModule                         4.99627       5.80282       6.8681        5.49236      0.675247        10     
reco:wclsdatasp:WireCellToolkit                     38.8234       56.5416       69.9146       59.731        9.29746        10     
reco:gaushit:GausHitFinder                          1.2034        1.65216       2.2732        1.64796      0.300358        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000778736    0.0016779    0.00231519    0.00178515    0.000515223      10     
reco:reco3d:SpacePointSolver                        15.7529       26.8885       66.1527       22.6953       13.8667        10     
reco:hitpdune:DisambigFromSpacePoints               1.67616       2.49259       4.71158       2.33158      0.862668        10     
reco:pandora:StandardPandora                        40.152        69.757        155.794       59.1027       33.5339        10     
reco:pandoraWriter:StandardPandora                 0.663871      0.872703       1.20467      0.845205      0.150912        10     
reco:pandoraTrack:LArPandoraTrackCreation           12.9911       22.6178       43.0757       22.5479       8.53524        10     
reco:pandoraShower:LArPandoraShowerCreation         13.6206       27.6046       53.7001       26.2188       12.1104        10     
reco:pandoracalo:Calorimetry                        5.64245       9.95953       17.5129       10.0205       3.39937        10     
reco:pandoracalonosce:Calorimetry                   5.55569       9.32403       15.8953       9.17517       3.04115        10     
reco:pandorapid:Chi2ParticleID                    0.00564302    0.00784497     0.0108528    0.00737156    0.00169785       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0477959     0.0644307     0.0864526     0.0630087     0.0126408       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00548313    0.00752741     0.0105952    0.00752271    0.00135614       10     
reco:pandoraShowercalo:ShowerCalorimetry            12.9114       21.5526       36.1106       20.1373       7.32574        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       11.3176       22.774        43.7786       19.9077       10.1018        10     
reco:emtrkmichelid:EmTrackMichelId                  182.948       229.385       305.108       223.453       34.7663        10     
reco:crttag:SingleCRTMatchingProducer              0.032103      0.0503763     0.0917251     0.0494171     0.0164757       10     
reco:crtreco:TwoCRTMatchingProducer                0.0300915     0.0483888     0.0839022     0.0476862     0.0147784       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.824292       1.13205       1.48828       1.1328       0.226979        10     
reco:pandora2Track:LArPandoraTrackCreation          47.6096       76.5722       111.647       68.6291       23.0612        10     
reco:pandora2calo:Calorimetry                       13.3165       34.1996       54.6672       30.8678       12.5672        10     
reco:pandora2calonosce:Calorimetry                  13.9113       33.4591       60.031        29.4368       13.3692        10     
reco:pandora2pid:Chi2ParticleID                    0.0116524     0.0191344     0.0275919     0.0182821     0.0046798       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0661903     0.0892886     0.127975      0.0827436     0.0199953       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0131944     0.0193479     0.0304243     0.0182737    0.00472575       10     
reco:pandora2Shower:LArPandoraShowerCreation        10.2471       23.3067       38.6558       20.7421       8.06784        10     
reco:pandora2Showercalo:ShowerCalorimetry           11.9283       27.109        47.4027       22.0342       10.5665        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      9.32654       21.9704       35.4999       20.0316       7.84112        10     
[art]:TriggerResults:TriggerResultInserter         3.46e-05     6.41339e-05   0.000129955    5.328e-05    2.76288e-05      10     
end_path:out1:RootOutput                           7.461e-06    1.14113e-05    1.98e-05     1.1157e-05     3.583e-06       10     
end_path:out1:RootOutput(write)                     3.11577       3.92917       5.70088       3.61944      0.762159        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5621.1 MB
  Peak resident set size usage (VmHWM): 4817.66 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.27 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.55 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.15 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 12598436
-rw-r--r--. 1 dune009 dune 2872829344 Jan 20 19:42 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2728712167 Jan 20 19:39 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2635009077 Jan 20 21:46 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2324772868 Jan 20 16:17 g4_stage2.root
-rw-r--r--. 1 dune009 dune 2244526826 Jan 20 16:00 g4_stage1.root
-rw-r--r--. 1 dune009 dune   94332610 Jan 20 21:35 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     243408 Jan 20 21:46 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Jan 20 21:46 mem.db
-rw-r--r--. 1 dune009 dune      62376 Jan 20 21:46 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Jan 20 21:46 time.db
-rw-r--r--. 1 dune009 dune        519 Jan 20 19:42 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 20 16:17 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Jan 20 21:46 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Jan 20 15:52 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Jan 20 21:46 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Jan 20 19:51 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Jan 20 19:42 debugprod.log
justIN time: 2026-02-04 07:23:52 UTC       justIN version: 01.06.00