justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 20847.0@dunegpschedd01.fnal.gov

Jobsub ID20847.0@dunegpschedd01.fnal.gov
Workflow ID378
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-08-04 18:10:09
SiteCZ_FZU
EntryNova_CZ_FZU_osgce2
Last heartbeat2025-08-04 22:47:28
From worker nodeHostnamemahagon03.farm.particle.cz
cpuinfoAMD EPYC 7301 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-04 18:19:19
Input filesmonte-carlo-000378-000001
JobscriptExit code0
Real time4h (16038s)
CPU time4h (15479s = 96%)
Max RSS bytes3972440064 (3788 MiB)
Outputting started2025-08-04 22:46:39
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/00378/1/001/reco_stage1.root
Finished2025-08-04 22:47:28
Saved logsjustin-logs:20847.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

gitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 125534 traces, 0 summary
Retagger: tagging trace set: gauss with 96547 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 96547 traces tagged "gauss"
FrameSaver: q=3.43845e+06 n=1749132 tag=gauss
wclsFrameSaver: saving 125534 traces tagged "wiener"
FrameSaver: q=3.62139e+06 n=1557807 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 31962 XUs and 26507 XVs -> 275 XUVs
C:0 T:1 47761 XUs and 41956 XVs -> 3136 XUVs
C:0 T:2 27047 XUs and 52226 XVs -> 1359 XUVs
C:0 T:3 16151 XUs and 27270 XVs -> 120 XUVs
C:0 T:4 11713 XUs and 13432 XVs -> 83 XUVs
C:0 T:5 43763 XUs and 36627 XVs -> 3598 XUVs
C:0 T:6 101613 XUs and 119953 XVs -> 3347 XUVs
C:0 T:7 40097 XUs and 45377 XVs -> 373 XUVs
C:0 T:8 7583 XUs and 7332 XVs -> 48 XUVs
C:0 T:9 55177 XUs and 65580 XVs -> 5177 XUVs
C:0 T:10 78320 XUs and 63565 XVs -> 7916 XUVs
C:0 T:11 215 XUs and 148 XVs -> 36 XUVs
25468 XUVs total
11319 collection wire objects
23936 potential space points
Neighbour search...
1507174 tests to find 739758 neighbours
Iterating with no regularization...
Begin: 8.97675e+07
0 7.45519e+07
1 7.37807e+07
2 7.3686e+07
3 7.36596e+07
Now with regularization...
Begin: 6.39319e+07
0 6.38758e+07
PcaShowerParticleBuildingAlgorithm::OpeningAngle - principal eigenvalue less than or equal to 0.
run: 1 ; subrun: 0 ; event: 10
evttime: 408461909
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 408461909
fLifetime: 30000 [us]
05-Aug-2025 00:46:36 CEST  Closed output file "reco_stage1.root"
05-Aug-2025 00:46:36 CEST  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          442.01        562.739       681.032       564.64        80.2186        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00223105    0.00459463    0.00641481    0.00491403    0.00143869       10     
reco:ophit:OpHitFinder                             0.360058       0.41325      0.485218      0.395867      0.040906        10     
reco:opflash:OpFlashFinder                         0.188653      0.212984      0.239762      0.211421      0.0162554       10     
reco:opslicer:OpSlicer                             0.900004       1.05633       1.52585       1.00825       0.17631        10     
reco:caldata:DataPrepModule                         8.51434       8.98195       9.73771       8.95369      0.310661        10     
reco:wclsdatasp:WireCellToolkit                     59.2056       62.1431       68.2167       61.5892       2.27483        10     
reco:gaushit:GausHitFinder                          1.48148       1.86677       2.17119       1.87373      0.221486        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00059279    0.000716526   0.000972938   0.000692389   9.80207e-05      10     
reco:reco3d:SpacePointSolver                        9.30603       18.5863       34.1767       16.6518       7.68664        10     
reco:hitpdune:DisambigFromSpacePoints               1.55761       2.31639       3.12599       2.27384      0.489799        10     
reco:pandora:StandardPandora                        34.2168       47.2919        61.54        45.6213       10.0549        10     
reco:pandoraWriter:StandardPandora                 0.534016      0.695804      0.875011      0.693094      0.0847884       10     
reco:pandoraTrack:LArPandoraTrackCreation           4.69202       6.36307       9.85141       6.30791       1.43716        10     
reco:pandoraShower:LArPandoraShowerCreation         3.6669        6.89048       9.72029       6.98292       1.9541         10     
reco:pandoracalo:Calorimetry                        2.45942       3.55659       4.91637       3.66726      0.794787        10     
reco:pandoracalonosce:Calorimetry                   2.17157       3.2615        4.4746        3.40277      0.761296        10     
reco:pandorapid:Chi2ParticleID                    0.00813372     0.010059      0.0134882    0.00986594    0.00150259       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0735554     0.0987508     0.145877      0.0929994     0.0193361       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00827144    0.00986817     0.0114295    0.00947836    0.00107806       10     
reco:pandoraShowercalo:ShowerCalorimetry            3.04562       7.10057       10.3854       7.27362       2.21554        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.87892       6.95463       10.0317       7.14403       2.26107        10     
reco:emtrkmichelid:EmTrackMichelId                  261.74        316.153       365.181       318.756       39.7378        10     
reco:crttag:SingleCRTMatchingProducer              0.0158806     0.025166      0.0357978     0.023812     0.00703184       10     
reco:crtreco:TwoCRTMatchingProducer                0.0122476     0.0162723     0.0223323     0.0155989    0.00321047       10     
reco:anodepiercerst0:T0RecoAnodePiercers            1.08461       1.36487       1.73713        1.345       0.222099        10     
reco:pandora2Track:LArPandoraTrackCreation          11.3793       19.5324       27.1302       20.1358       4.79314        10     
reco:pandora2calo:Calorimetry                       5.14146       9.98887       14.0636       10.2963       2.76765        10     
reco:pandora2calonosce:Calorimetry                  4.89572       9.26842       13.5608       9.49727       2.67483        10     
reco:pandora2pid:Chi2ParticleID                    0.0177884     0.0243741     0.0286475     0.0252589    0.00316069       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0889878     0.115977      0.136843      0.118403      0.0145693       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0176074     0.0236607     0.0292226     0.0238972    0.00322768       10     
reco:pandora2Shower:LArPandoraShowerCreation        4.33458       7.32683       10.2845       7.54484       1.77171        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.3911        8.67479       12.3887       9.33253       2.52895        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.90821       7.70123       11.0307       8.42205       2.21475        10     
[art]:TriggerResults:TriggerResultInserter        7.4801e-05    0.000105411   0.000232258   9.36615e-05   4.33731e-05      10     
end_path:out1:RootOutput                          1.3075e-05    1.76313e-05   2.9867e-05    1.6872e-05    4.48175e-06      10     
end_path:out1:RootOutput(write)                     3.48653       4.55687       5.63552       4.68615      0.618499        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4798 MB
  Peak resident set size usage (VmHWM): 3972.44 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.13 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.35 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   5.50 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.22 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.09 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.01 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 9428500
-rw-r--r-- 1 fermilab users 2228795942 Aug  4 23:09 detsim_stage2.root
-rw-r--r-- 1 fermilab users 2116620599 Aug  4 23:06 detsim_stage1.root
-rw-r--r-- 1 fermilab users 1884556335 Aug  5 00:46 reco_stage1.root
-rw-r--r-- 1 fermilab users 1714676864 Aug  4 20:42 g4_stage2.root
-rw-r--r-- 1 fermilab users 1641840621 Aug  4 20:26 g4_stage1.root
-rw-r--r-- 1 fermilab users   67663000 Aug  5 00:38 Pandora_Events.pndr
-rw-r--r-- 1 fermilab users     243028 Aug  5 00:46 jobscript.log
-rw-r--r-- 1 fermilab users     217088 Aug  5 00:46 mem.db
-rw-r--r-- 1 fermilab users      62376 Aug  5 00:46 Pandora_Geometry.xml
-rw-r--r-- 1 fermilab users      36864 Aug  5 00:46 time.db
-rw-r--r-- 1 fermilab users        519 Aug  4 23:09 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 fermilab users        519 Aug  4 20:42 g4_protoDUNE_hist.root
-rw-r--r-- 1 fermilab users        519 Aug  5 00:46 protoDUNE_reco1_hist.root
-rw-r--r-- 1 fermilab users         52 Aug  4 20:19 all-input-dids.txt
-rw-r--r-- 1 fermilab users          7 Aug  5 00:46 justin-processed-pfns.txt
-rw-r--r-- 1 fermilab users          0 Aug  4 23:17 ProtoDUNE.log
-rw-r--r-- 1 fermilab users          0 Aug  4 23:09 debugprod.log
justIN time: 2025-08-04 23:59:08 UTC       justIN version: 01.04.00