justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 14072.0@dunegpschedd02.fnal.gov

Jobsub ID14072.0@dunegpschedd02.fnal.gov
Workflow ID258
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-08-02 23:05:48
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-08-03 02:26:49
From worker nodeHostnamewn-db-04.gina.surf.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-02 23:06:36
Input filesmonte-carlo-000258-000001
JobscriptExit code0
Real time3h (11916s)
CPU time3h (11694s = 98%)
Max RSS bytes4009889792 (3824 MiB)
Outputting started2025-08-03 02:25:13
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/00258/1/001/reco_stage1.root
Finished2025-08-03 02:26:44
Saved logsjustin-logs:14072.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ning tool adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 111939 traces, 0 summary
Retagger: tagging trace set: gauss with 89651 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 89651 traces tagged "gauss"
FrameSaver: q=3.05499e+06 n=1617317 tag=gauss
wclsFrameSaver: saving 111939 traces tagged "wiener"
FrameSaver: q=3.22005e+06 n=1461712 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 49262 XUs and 36068 XVs -> 855 XUVs
C:0 T:1 97579 XUs and 73138 XVs -> 15092 XUVs
C:0 T:2 103770 XUs and 132525 XVs -> 6557 XUVs
C:0 T:3 49042 XUs and 54622 XVs -> 333 XUVs
C:0 T:4 8310 XUs and 7201 XVs -> 100 XUVs
C:0 T:5 26321 XUs and 17613 XVs -> 1691 XUVs
C:0 T:6 55702 XUs and 84395 XVs -> 2477 XUVs
C:0 T:7 23392 XUs and 37438 XVs -> 254 XUVs
C:0 T:8 7640 XUs and 7025 XVs -> 82 XUVs
C:0 T:9 42440 XUs and 52362 XVs -> 3096 XUVs
C:0 T:10 14718 XUs and 7773 XVs -> 1549 XUVs
C:0 T:11 45 XUs and 20 XVs -> 3 XUVs
32089 XUVs total
9286 collection wire objects
29059 potential space points
Neighbour search...
3298591 tests to find 1653966 neighbours
Iterating with no regularization...
Begin: 6.08663e+07
0 5.338e+07
1 5.29158e+07
2 5.28433e+07
3 5.28206e+07
Now with regularization...
Begin: 4.48938e+07
0 4.48621e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 408425526
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 408425526
fLifetime: 30000 [us]
03-Aug-2025 04:25:12 CEST  Closed output file "reco_stage1.root"
03-Aug-2025 04:25:12 CEST  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          301.097       395.173       592.366       352.41        91.9296        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00122088    0.00455023    0.00846451    0.00442408    0.00214222       10     
reco:ophit:OpHitFinder                             0.222569      0.264005      0.299222      0.259284      0.0234954       10     
reco:opflash:OpFlashFinder                         0.111584      0.154356      0.190038      0.151562      0.0264156       10     
reco:opslicer:OpSlicer                              0.49859      0.655424      0.824957      0.611129      0.109963        10     
reco:caldata:DataPrepModule                         4.80999       5.02944       5.43849       5.00073      0.181979        10     
reco:wclsdatasp:WireCellToolkit                     39.0798       61.7878       73.7497       67.2995       11.5774        10     
reco:gaushit:GausHitFinder                         0.947605       1.26766       1.63113       1.29256      0.188123        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00045737    0.000651311   0.000941822   0.000627304   0.000160473      10     
reco:reco3d:SpacePointSolver                        5.57765       16.1719       41.3094       10.7465       12.6822        10     
reco:hitpdune:DisambigFromSpacePoints              0.846153       1.44351       2.2639        1.33014      0.409683        10     
reco:pandora:StandardPandora                        19.9455       33.4492       74.4336       27.3125       15.9639        10     
reco:pandoraWriter:StandardPandora                 0.297347      0.409784      0.528462      0.405282      0.0696031       10     
reco:pandoraTrack:LArPandoraTrackCreation           2.62023       4.43427       6.68315       4.09326       1.3193         10     
reco:pandoraShower:LArPandoraShowerCreation         2.80448       5.3145        10.7409       4.24106       2.70459        10     
reco:pandoracalo:Calorimetry                        1.75148       2.50334       3.60445       2.35234      0.624175        10     
reco:pandoracalonosce:Calorimetry                   1.40652       2.39972       4.10601       2.12294      0.811796        10     
reco:pandorapid:Chi2ParticleID                    0.00475704    0.00573854    0.00651631    0.00565467    0.000590708      10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0389875     0.0481193     0.0630765     0.0454197    0.00734275       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID           0.0045491    0.00555865    0.00701796    0.00536826    0.000670727      10     
reco:pandoraShowercalo:ShowerCalorimetry            2.77758       5.16513       10.6544       4.36526       2.41804        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.50676       4.92641       9.87961       4.22535       2.31008        10     
reco:emtrkmichelid:EmTrackMichelId                  152.746       201.662       267.109       193.31        33.7507        10     
reco:crttag:SingleCRTMatchingProducer             0.00774628     0.0137425     0.0217817     0.0122759    0.00452356       10     
reco:crtreco:TwoCRTMatchingProducer               0.00805412     0.0116316     0.0209086     0.0100063    0.00390006       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.651491      0.805103       1.13588      0.752171      0.131599        10     
reco:pandora2Track:LArPandoraTrackCreation          8.03655       14.0972       25.769        11.9233       5.7064         10     
reco:pandora2calo:Calorimetry                       3.57192       7.06962       13.1663       6.33809       2.91655        10     
reco:pandora2calonosce:Calorimetry                  3.35176       6.63733       12.2561       5.57428       2.84552        10     
reco:pandora2pid:Chi2ParticleID                    0.0107392     0.0136169     0.0179534     0.0129936    0.00244077       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0505473     0.0620396     0.0838545     0.0590512     0.010417        10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0111459     0.0139113     0.0192149     0.0133124    0.00244694       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.85311       5.13077       8.60844       4.57642       1.89381        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.25331       5.92479       10.592        5.26116       2.35136        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.96524       5.28503       9.34607       4.67748       2.08576        10     
[art]:TriggerResults:TriggerResultInserter        3.0066e-05    4.92426e-05   0.000105421   4.03665e-05   2.10076e-05      10     
end_path:out1:RootOutput                           5.831e-06    7.1908e-06    1.1171e-05    6.8025e-06    1.40532e-06      10     
end_path:out1:RootOutput(write)                     2.07955       2.8498        3.95491       2.7263       0.543275        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4813.55 MB
  Peak resident set size usage (VmHWM): 4009.89 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.22 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.11 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.12 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.05 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9314352
-rw-r--r--. 1 dune003 dune 2223418905 Aug  3 03:17 detsim_stage2.root
-rw-r--r--. 1 dune003 dune 2110942468 Aug  3 03:15 detsim_stage1.root
-rw-r--r--. 1 dune003 dune 1900289496 Aug  3 04:25 reco_stage1.root
-rw-r--r--. 1 dune003 dune 1645302920 Aug  3 01:23 g4_stage2.root
-rw-r--r--. 1 dune003 dune 1585079906 Aug  3 01:12 g4_stage1.root
-rw-r--r--. 1 dune003 dune   72252090 Aug  3 04:21 Pandora_Events.pndr
-rw-r--r--. 1 dune003 dune     243202 Aug  3 04:25 jobscript.log
-rw-r--r--. 1 dune003 dune     217088 Aug  3 04:25 mem.db
-rw-r--r--. 1 dune003 dune      62376 Aug  3 04:25 Pandora_Geometry.xml
-rw-r--r--. 1 dune003 dune      36864 Aug  3 04:25 time.db
-rw-r--r--. 1 dune003 dune        519 Aug  3 03:17 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune003 dune        519 Aug  3 01:23 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune003 dune        519 Aug  3 04:25 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune003 dune         52 Aug  3 01:06 all-input-dids.txt
-rw-r--r--. 1 dune003 dune          7 Aug  3 04:25 justin-processed-pfns.txt
-rw-r--r--. 1 dune003 dune          0 Aug  3 03:23 ProtoDUNE.log
-rw-r--r--. 1 dune003 dune          0 Aug  3 03:17 debugprod.log
justIN time: 2025-08-04 16:02:35 UTC       justIN version: 01.04.00