justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 279709.0@dunegpschedd01.fnal.gov

Jobsub ID279709.0@dunegpschedd01.fnal.gov
Workflow ID11990
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-17 01:13:48
SiteUS_BNL
EntryDUNE_US_BNL_spoolce01
Last heartbeat2026-01-17 06:03:53
From worker nodeHostnamespool1504.sdcc.bnl.gov
cpuinfoIntel(R) Xeon(R) Gold 6336Y CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-17 01:15:11
Input filesmonte-carlo-011990-000001
JobscriptExit code0
Real time4h (17289s)
CPU time4h (17044s = 98%)
Max RSS bytes4438585344 (4232 MiB)
Outputting started2026-01-17 06:03:20
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11990/1/001/reco_stage1.root
Finished2026-01-17 06:03:53
Saved logsjustin-logs:279709.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Adc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 142333 traces, 0 summary
Retagger: tagging trace set: gauss with 108517 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 108517 traces tagged "gauss"
FrameSaver: q=3.94612e+06 n=1970629 tag=gauss
wclsFrameSaver: saving 142333 traces tagged "wiener"
FrameSaver: q=4.19042e+06 n=1768403 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 54846 XUs and 49407 XVs -> 3961 XUVs
C:0 T:1 126061 XUs and 108776 XVs -> 18130 XUVs
C:0 T:2 68279 XUs and 70420 XVs -> 2997 XUVs
C:0 T:3 36527 XUs and 32770 XVs -> 192 XUVs
C:0 T:4 12836 XUs and 7349 XVs -> 66 XUVs
C:0 T:5 45125 XUs and 22341 XVs -> 6043 XUVs
C:0 T:6 137441 XUs and 170715 XVs -> 13659 XUVs
C:0 T:7 55709 XUs and 66644 XVs -> 1304 XUVs
C:0 T:8 14319 XUs and 10073 XVs -> 119 XUVs
C:0 T:9 100570 XUs and 88210 XVs -> 8114 XUVs
C:0 T:10 23733 XUs and 14699 XVs -> 2969 XUVs
C:0 T:11 254 XUs and 172 XVs -> 21 XUVs
57575 XUVs total
11674 collection wire objects
48263 potential space points
Neighbour search...
5727627 tests to find 2992526 neighbours
Iterating with no regularization...
Begin: 6.76873e+07
0 5.60071e+07
1 5.49442e+07
2 5.47863e+07
3 5.47392e+07
Now with regularization...
Begin: 4.34655e+07
0 4.3422e+07
1 4.33797e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411783868
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411783868
fLifetime: 30000 [us]
17-Jan-2026 01:03:18 EST  Closed output file "reco_stage1.root"
17-Jan-2026 01:03:18 EST  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          556.309       752.086       1140.07       727.643       159.235        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00157961     0.0052304    0.00914378     0.0051773    0.00209742       10     
reco:ophit:OpHitFinder                             0.274201      0.427317      0.548703      0.434334      0.0715509       10     
reco:opflash:OpFlashFinder                         0.181812      0.293806      0.436479      0.297962      0.0653511       10     
reco:opslicer:OpSlicer                             0.558859      0.915035       1.10459      0.960702      0.171845        10     
reco:caldata:DataPrepModule                         7.95632       8.75185       9.21451       8.89404      0.427133        10     
reco:wclsdatasp:WireCellToolkit                     59.9336       63.7501       66.642        63.4126       1.96263        10     
reco:gaushit:GausHitFinder                          1.82299       2.43172       3.42456       2.32965      0.457078        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000874261   0.00129529    0.00153221    0.00132588    0.000206964      10     
reco:reco3d:SpacePointSolver                        13.0172       25.3052       44.9084        24.2         8.60926        10     
reco:hitpdune:DisambigFromSpacePoints               1.96762       3.35531       6.05157       2.87661       1.17995        10     
reco:pandora:StandardPandora                        39.4089       59.6912       117.28        52.7186       21.2555        10     
reco:pandoraWriter:StandardPandora                 0.526462      0.782814       1.21652      0.721452      0.173045        10     
reco:pandoraTrack:LArPandoraTrackCreation           11.8879       18.9314       27.0977       18.4375       4.17003        10     
reco:pandoraShower:LArPandoraShowerCreation         14.4398       21.8123       37.6481       20.5709       6.80236        10     
reco:pandoracalo:Calorimetry                        5.5511        8.60692       12.9012       8.48128       1.92253        10     
reco:pandoracalonosce:Calorimetry                   5.15651       8.20237       11.6335       8.21686       1.82347        10     
reco:pandorapid:Chi2ParticleID                    0.00793056     0.0115183     0.0166301     0.0114126    0.00228692       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0644082     0.080234      0.102876      0.0793635    0.00988748       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00782366     0.0105422     0.0131478     0.0103869    0.00145632       10     
reco:pandoraShowercalo:ShowerCalorimetry            10.5243       18.9921       34.1035       17.8139       6.83564        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       11.5435       18.5507       32.9583       17.0436       6.43528        10     
reco:emtrkmichelid:EmTrackMichelId                  255.318       321.68        452.021       319.28        52.9125        10     
reco:crttag:SingleCRTMatchingProducer              0.0354886     0.0478189     0.062497      0.0470769    0.00791742       10     
reco:crtreco:TwoCRTMatchingProducer                0.0318012     0.0415506     0.056846      0.0421787    0.00707335       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.13053       1.51736       2.33585       1.40156      0.375968        10     
reco:pandora2Track:LArPandoraTrackCreation          36.7564       58.5125       101.417       57.5778       18.3579        10     
reco:pandora2calo:Calorimetry                       14.455        25.6797       45.634        25.2109       8.60846        10     
reco:pandora2calonosce:Calorimetry                  13.3334       23.6534       38.8992       23.4267       7.54994        10     
reco:pandora2pid:Chi2ParticleID                    0.0216479     0.0252188     0.0343105     0.0249071    0.00363968       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0813022     0.102353      0.144716      0.0970839     0.0165394       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0210474     0.0248725     0.0318577     0.0248196    0.00342139       10     
reco:pandora2Shower:LArPandoraShowerCreation        11.4063       17.8316       28.8316       17.1104       5.27263        10     
reco:pandora2Showercalo:ShowerCalorimetry           11.6879       20.4903       31.1397       19.8113       6.02123        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      9.67047       17.0767       26.7516       16.4438       5.3889         10     
[art]:TriggerResults:TriggerResultInserter        8.5921e-05    0.000100361   0.00013874    9.3776e-05    1.55309e-05      10     
end_path:out1:RootOutput                           1.306e-05    1.44362e-05   1.5881e-05    1.44855e-05   7.97904e-07      10     
end_path:out1:RootOutput(write)                     3.5944        4.23065       5.87985       4.01795       0.60965        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5263.27 MB
  Peak resident set size usage (VmHWM): 4438.59 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.13 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.37 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.54 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.31 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.08 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11043700
-rw-r--r-- 1 fermipro fermilab 2554143388 Jan 16 22:54 detsim_stage2.root
-rw-r--r-- 1 fermipro fermilab 2427373015 Jan 16 22:51 detsim_stage1.root
-rw-r--r-- 1 fermipro fermilab 2261524837 Jan 17 01:03 reco_stage1.root
-rw-r--r-- 1 fermipro fermilab 2021295555 Jan 16 20:44 g4_stage2.root
-rw-r--r-- 1 fermipro fermilab 1963891179 Jan 16 20:23 g4_stage1.root
-rw-r--r-- 1 fermipro fermilab   79887190 Jan 17 00:52 Pandora_Events.pndr
-rw-r--r-- 1 fermipro fermilab     242794 Jan 17 01:03 jobscript.log
-rw-r--r-- 1 fermipro fermilab     217088 Jan 17 01:03 mem.db
-rw-r--r-- 1 fermipro fermilab      62376 Jan 17 01:03 Pandora_Geometry.xml
-rw-r--r-- 1 fermipro fermilab      36864 Jan 17 01:03 time.db
-rw-r--r-- 1 fermipro fermilab        519 Jan 16 22:54 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 fermipro fermilab        519 Jan 16 20:44 g4_protoDUNE_hist.root
-rw-r--r-- 1 fermipro fermilab        519 Jan 17 01:03 protoDUNE_reco1_hist.root
-rw-r--r-- 1 fermipro fermilab         52 Jan 16 20:15 all-input-dids.txt
-rw-r--r-- 1 fermipro fermilab          7 Jan 17 01:03 justin-processed-pfns.txt
-rw-r--r-- 1 fermipro fermilab          0 Jan 16 23:04 ProtoDUNE.log
-rw-r--r-- 1 fermipro fermilab          0 Jan 16 22:54 debugprod.log
justIN time: 2026-02-04 04:33:20 UTC       justIN version: 01.06.00