justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 12687.1@dunegpschedd02.fnal.gov

Jobsub ID12687.1@dunegpschedd02.fnal.gov
Workflow ID148
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-07-30 08:29:23
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-07-30 12:29:34
From worker nodeHostnamewn-lb-16.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-07-30 08:30:24
Input filesmonte-carlo-000148-000001
JobscriptExit code0
Real time3h (14226s)
CPU time3h (13934s = 97%)
Max RSS bytes4162510848 (3969 MiB)
Outputting started2025-07-30 12:27:31
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/00148/1/001/reco_stage1.root
Finished2025-07-30 12:29:34
Saved logsjustin-logs:12687.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

oolBasedRawDigitPrepService:prepare:   Running tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 86080 traces, 0 summary
Retagger: tagging trace set: gauss with 66203 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 66203 traces tagged "gauss"
FrameSaver: q=2.13069e+06 n=1162232 tag=gauss
wclsFrameSaver: saving 86080 traces tagged "wiener"
FrameSaver: q=2.24565e+06 n=1022446 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 16683 XUs and 13395 XVs -> 142 XUVs
C:0 T:1 38631 XUs and 22530 XVs -> 1785 XUVs
C:0 T:2 62080 XUs and 97331 XVs -> 3677 XUVs
C:0 T:3 32855 XUs and 40683 XVs -> 271 XUVs
C:0 T:4 10002 XUs and 10337 XVs -> 99 XUVs
C:0 T:5 35245 XUs and 29187 XVs -> 3206 XUVs
C:0 T:6 75906 XUs and 116294 XVs -> 8351 XUVs
C:0 T:7 29362 XUs and 43777 XVs -> 462 XUVs
C:0 T:9 11393 XUs and 12718 XVs -> 797 XUVs
C:0 T:10 24628 XUs and 8073 XVs -> 2216 XUVs
C:0 T:11 385 XUs and 318 XVs -> 49 XUVs
21055 XUVs total
7669 collection wire objects
19435 potential space points
Neighbour search...
1297283 tests to find 663614 neighbours
Iterating with no regularization...
Begin: 4.24008e+07
0 3.32033e+07
1 3.27826e+07
2 3.27158e+07
3 3.26934e+07
Now with regularization...
Begin: 2.72441e+07
0 2.7226e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 408352929
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 408352929
fLifetime: 30000 [us]
30-Jul-2025 14:27:29 CEST  Closed output file "reco_stage1.root"
30-Jul-2025 14:27:29 CEST  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          253.253       375.389       493.561       377.537       79.0012        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00112685    0.00582153    0.00947527     0.0062294    0.00248781       10     
reco:ophit:OpHitFinder                             0.311639      0.364646      0.433002      0.355905      0.035877        10     
reco:opflash:OpFlashFinder                         0.156898      0.212275      0.261447      0.206376      0.0325737       10     
reco:opslicer:OpSlicer                             0.594413      0.761737       1.04013      0.777726      0.131103        10     
reco:caldata:DataPrepModule                         6.2328        6.46603       7.32756       6.33682      0.309545        10     
reco:wclsdatasp:WireCellToolkit                     47.8388       56.4881       67.2382       55.4668       6.92302        10     
reco:gaushit:GausHitFinder                         0.934848       1.24869       1.59766       1.24225      0.211671        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000362502   0.000587124   0.000771865   0.000613878   0.000137279      10     
reco:reco3d:SpacePointSolver                        7.63234       17.4715       29.0344       16.5041       7.73329        10     
reco:hitpdune:DisambigFromSpacePoints              0.818275       1.35462       2.01452       1.25655      0.448737        10     
reco:pandora:StandardPandora                        14.9792       28.6819       50.8073       27.3459       10.5698        10     
reco:pandoraWriter:StandardPandora                 0.319582      0.452986      0.594043      0.443607      0.0905939       10     
reco:pandoraTrack:LArPandoraTrackCreation           2.34345       5.85168       8.80383       5.63456       2.03536        10     
reco:pandoraShower:LArPandoraShowerCreation         1.97492       6.1818        10.9027       5.6634        2.97279        10     
reco:pandoracalo:Calorimetry                        1.27638       2.95719       4.18358       3.01539       1.02233        10     
reco:pandoracalonosce:Calorimetry                   1.15815       2.80356       4.05902       2.90792      0.973606        10     
reco:pandorapid:Chi2ParticleID                    0.00434398     0.0057427    0.00776543    0.00573426    0.000939524      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0370549     0.0487072     0.0623492     0.045606     0.00875387       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00443015    0.00554325    0.00656263    0.00571273    0.000675256      10     
reco:pandoraShowercalo:ShowerCalorimetry            1.94993       5.98678       10.3552       5.71124       2.78136        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       1.85369       5.58959       9.95287       5.35762       2.67972        10     
reco:emtrkmichelid:EmTrackMichelId                  134.368       174.98        224.944       170.229       31.2641        10     
reco:crttag:SingleCRTMatchingProducer             0.00955801     0.014918      0.0313343     0.0132813    0.00632684       10     
reco:crtreco:TwoCRTMatchingProducer                0.0079303     0.0131004     0.0184661     0.0126137    0.00418123       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.561442      0.955894       1.22514      0.983806      0.211215        10     
reco:pandora2Track:LArPandoraTrackCreation          7.25596       16.8784       25.7209       16.6287       6.27626        10     
reco:pandora2calo:Calorimetry                       3.42676       8.45289       13.2318       8.53298       3.57714        10     
reco:pandora2calonosce:Calorimetry                  3.30142       8.11358       13.1248       8.07171       3.30661        10     
reco:pandora2pid:Chi2ParticleID                   0.00932151     0.0124178     0.0159767     0.0126679     0.0019399       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0469918     0.063408      0.0890889     0.0617563     0.0125959       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0096608     0.0127047     0.015424      0.0130342    0.00182542       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.97993       6.46469       10.4456       5.8928        2.52221        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.13937       6.9703        11.5545       6.32023       3.07398        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.89044       6.25681       11.0138       5.71068        2.837         10     
[art]:TriggerResults:TriggerResultInserter        4.0781e-05    6.78968e-05   0.000148261   6.3651e-05    2.93564e-05      10     
end_path:out1:RootOutput                           7.361e-06    9.3259e-06    1.7546e-05    8.3825e-06    2.79704e-06      10     
end_path:out1:RootOutput(write)                     2.23696       3.02899       4.04385       2.91476      0.629293        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4967.09 MB
  Peak resident set size usage (VmHWM): 4162.51 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.29 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.99 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.17 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9793892
-rw-r--r--. 1 dune003 dune 2309305639 Jul 30 13:22 detsim_stage2.root
-rw-r--r--. 1 dune003 dune 2201284979 Jul 30 13:20 detsim_stage1.root
-rw-r--r--. 1 dune003 dune 1959158770 Jul 30 14:27 reco_stage1.root
-rw-r--r--. 1 dune003 dune 1784729125 Jul 30 10:53 g4_stage2.root
-rw-r--r--. 1 dune003 dune 1707856215 Jul 30 10:37 g4_stage1.root
-rw-r--r--. 1 dune003 dune   65990670 Jul 30 14:24 Pandora_Events.pndr
-rw-r--r--. 1 dune003 dune     242998 Jul 30 14:27 jobscript.log
-rw-r--r--. 1 dune003 dune     217088 Jul 30 14:27 mem.db
-rw-r--r--. 1 dune003 dune      62376 Jul 30 14:27 Pandora_Geometry.xml
-rw-r--r--. 1 dune003 dune      36864 Jul 30 14:27 time.db
-rw-r--r--. 1 dune003 dune        519 Jul 30 13:22 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune003 dune        519 Jul 30 10:53 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune003 dune        519 Jul 30 14:27 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune003 dune         52 Jul 30 10:30 all-input-dids.txt
-rw-r--r--. 1 dune003 dune          7 Jul 30 14:27 justin-processed-pfns.txt
-rw-r--r--. 1 dune003 dune          0 Jul 30 13:30 ProtoDUNE.log
-rw-r--r--. 1 dune003 dune          0 Jul 30 13:22 debugprod.log
justIN time: 2025-08-05 06:39:27 UTC       justIN version: 01.04.00