justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 260606.0@dunegpschedd02.fnal.gov

Jobsub ID260606.0@dunegpschedd02.fnal.gov
Workflow ID11143
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-12-10 23:21:29
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2025-12-11 03:47:34
From worker nodeHostnamewn-la-06.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-10 23:23:31
Input filesmonte-carlo-011143-000001
JobscriptExit code0
Real time4h (15742s)
CPU time4h (15575s = 98%)
Max RSS bytes4531552256 (4321 MiB)
Outputting started2025-12-11 03:45:54
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/11143/1/001/reco_stage1.root
Finished2025-12-11 03:47:34
Saved logsjustin-logs:260606.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 135971 traces, 0 summary
Retagger: tagging trace set: gauss with 99308 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 99308 traces tagged "gauss"
FrameSaver: q=3.39263e+06 n=1742007 tag=gauss
wclsFrameSaver: saving 135971 traces tagged "wiener"
FrameSaver: q=3.62058e+06 n=1528029 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 29488 XUs and 20949 XVs -> 330 XUVs
C:0 T:1 71451 XUs and 55497 XVs -> 7137 XUVs
C:0 T:2 89240 XUs and 114330 XVs -> 8266 XUVs
C:0 T:3 36721 XUs and 47597 XVs -> 528 XUVs
C:0 T:4 9449 XUs and 8422 XVs -> 146 XUVs
C:0 T:5 35101 XUs and 27704 XVs -> 2826 XUVs
C:0 T:6 163102 XUs and 110738 XVs -> 12088 XUVs
C:0 T:7 53464 XUs and 37207 XVs -> 549 XUVs
C:0 T:8 9876 XUs and 11866 XVs -> 114 XUVs
C:0 T:9 76319 XUs and 117912 XVs -> 6062 XUVs
C:0 T:10 50742 XUs and 41162 XVs -> 5986 XUVs
C:0 T:11 55 XUs and 16 XVs -> 2 XUVs
44034 XUVs total
13297 collection wire objects
40002 potential space points
Neighbour search...
4389022 tests to find 2174678 neighbours
Iterating with no regularization...
Begin: 5.75879e+07
0 4.59275e+07
1 4.50073e+07
2 4.48634e+07
3 4.48212e+07
Now with regularization...
Begin: 3.59813e+07
0 3.59291e+07
1 3.58979e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411041209
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411041209
fLifetime: 30000 [us]
11-Dec-2025 04:45:52 CET  Closed output file "reco_stage1.root"
11-Dec-2025 04:45:52 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          305.756       465.465       712.401       453.709       108.778        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00131809    0.00608286     0.0102522    0.00575216    0.00292819       10     
reco:ophit:OpHitFinder                             0.285833      0.361161      0.476097      0.339382      0.0721273       10     
reco:opflash:OpFlashFinder                         0.165939      0.257674      0.361471      0.249252      0.0717592       10     
reco:opslicer:OpSlicer                             0.531389      0.739965       1.00011      0.721916      0.156945        10     
reco:caldata:DataPrepModule                         4.84595       5.41773       6.9441        4.98442      0.736471        10     
reco:wclsdatasp:WireCellToolkit                     36.0797       44.3151       60.6659       45.1118       6.68788        10     
reco:gaushit:GausHitFinder                         0.929464       1.27675       1.76045       1.20292      0.275255        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00044462    0.000669591   0.000873657   0.000670045   0.000129396      10     
reco:reco3d:SpacePointSolver                        13.0948       22.8996       51.5814       19.7466       11.2145        10     
reco:hitpdune:DisambigFromSpacePoints              0.999492       1.78988       2.93304       1.76684      0.525066        10     
reco:pandora:StandardPandora                        27.142        45.9015       88.8879       35.599        19.2935        10     
reco:pandoraWriter:StandardPandora                 0.353799       0.51157      0.662236      0.512873      0.0817201       10     
reco:pandoraTrack:LArPandoraTrackCreation           4.55889        9.853        15.0907       9.84791       3.16804        10     
reco:pandoraShower:LArPandoraShowerCreation         6.51144       12.2918       22.9528       11.8065       4.37397        10     
reco:pandoracalo:Calorimetry                        2.71617       5.20188       7.70112       5.09574       1.5399         10     
reco:pandoracalonosce:Calorimetry                   2.45655       4.83175       7.94291       4.57848       1.69804        10     
reco:pandorapid:Chi2ParticleID                    0.00418944    0.00607478    0.00807868    0.00597728    0.00108265       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0386141     0.0532432     0.0727044     0.0509059    0.00999707       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00424864    0.00577976     0.007741     0.00533553    0.00107118       10     
reco:pandoraShowercalo:ShowerCalorimetry            6.63417       11.8157       21.5214       11.4816       4.11648        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       5.27768       11.0674       20.2954       10.4441       3.85666        10     
reco:emtrkmichelid:EmTrackMichelId                  141.03        191.879       245.501       184.728       31.2883        10     
reco:crttag:SingleCRTMatchingProducer              0.0157573     0.0222734     0.0304131     0.0212621    0.00549328       10     
reco:crtreco:TwoCRTMatchingProducer                0.0127814     0.0167823     0.0252986     0.0159084    0.00383766       10     
reco:anodepiercerst0:T0RecoAnodePiercers            0.68644      0.958973       1.34286      0.924899      0.185147        10     
reco:pandora2Track:LArPandoraTrackCreation          15.1243       31.4564       63.4727       28.6541       12.325         10     
reco:pandora2calo:Calorimetry                       7.43389       14.2675       25.7198       14.1297       4.47796        10     
reco:pandora2calonosce:Calorimetry                  6.43865       13.7558       23.9688       13.135        4.33055        10     
reco:pandora2pid:Chi2ParticleID                   0.00985683     0.0131833     0.0174146     0.0130273    0.00226436       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0519311     0.0683082     0.0908902     0.0660246     0.0113461       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0110393     0.0132715     0.0169576     0.0130753    0.00209222       10     
reco:pandora2Shower:LArPandoraShowerCreation        5.24413       10.5602       20.7708       10.211        4.02322        10     
reco:pandora2Showercalo:ShowerCalorimetry           5.68731       11.0798       19.2086       10.843        3.32851        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      5.04236       9.21528       15.1323       9.28365       2.48985        10     
[art]:TriggerResults:TriggerResultInserter        3.6895e-05    5.56783e-05   0.000128823   4.53525e-05   2.64806e-05      10     
end_path:out1:RootOutput                           7.192e-06    8.4638e-06     1.354e-05     7.937e-06    1.78504e-06      10     
end_path:out1:RootOutput(write)                     2.2545        3.27645       4.3012        3.27728      0.542887        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5333.98 MB
  Peak resident set size usage (VmHWM): 4531.55 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.25 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.35 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.14 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 11136276
-rw-r--r--. 1 dune009 dune 2582904063 Dec 11 03:26 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2453897186 Dec 11 03:24 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2315411216 Dec 11 04:45 reco_stage1.root
-rw-r--r--. 1 dune009 dune 2018010234 Dec 11 00:45 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1946973583 Dec 11 00:30 g4_stage1.root
-rw-r--r--. 1 dune009 dune   85739630 Dec 11 04:40 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242719 Dec 11 04:45 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Dec 11 04:45 mem.db
-rw-r--r--. 1 dune009 dune      62376 Dec 11 04:45 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Dec 11 04:45 time.db
-rw-r--r--. 1 dune009 dune        519 Dec 11 03:26 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Dec 11 00:45 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Dec 11 04:45 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Dec 11 00:23 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Dec 11 04:45 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Dec 11 03:32 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Dec 11 03:26 debugprod.log
justIN time: 2025-12-18 18:07:36 UTC       justIN version: 01.05.03