justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 282173.0@dunegpschedd01.fnal.gov

Jobsub ID282173.0@dunegpschedd01.fnal.gov
Workflow ID12098
Stage ID1
User namepmelas@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-01-19 14:19:11
SiteCZ_FZU
EntryNova_CZ_FZU_osgce2
Last heartbeat2026-01-19 19:18:09
From worker nodeHostnamelupin01.farm.particle.cz
cpuinfoAMD EPYC 7532 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-19 14:19:55
Input filesmonte-carlo-012098-000001
JobscriptExit code0
Real time4h (17842s)
CPU time4h (17158s = 96%)
Max RSS bytes4278579200 (4080 MiB)
Outputting started2026-01-19 19:17:19
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/12098/1/001/reco_stage1.root
Finished2026-01-19 19:18:09
Saved logsjustin-logs:282173.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

sedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 105549 traces, 0 summary
Retagger: tagging trace set: gauss with 82647 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 82647 traces tagged "gauss"
FrameSaver: q=2.88809e+06 n=1498526 tag=gauss
wclsFrameSaver: saving 105549 traces tagged "wiener"
FrameSaver: q=3.03943e+06 n=1344370 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 25484 XUs and 20813 XVs -> 331 XUVs
C:0 T:1 36218 XUs and 29280 XVs -> 1792 XUVs
C:0 T:2 96928 XUs and 134800 XVs -> 4322 XUVs
C:0 T:3 48883 XUs and 63558 XVs -> 412 XUVs
C:0 T:4 12229 XUs and 9940 XVs -> 180 XUVs
C:0 T:5 50791 XUs and 28894 XVs -> 6340 XUVs
C:0 T:6 60040 XUs and 77820 XVs -> 4228 XUVs
C:0 T:7 29749 XUs and 38938 XVs -> 352 XUVs
C:0 T:8 5899 XUs and 5532 XVs -> 97 XUVs
C:0 T:9 105948 XUs and 44062 XVs -> 7919 XUVs
C:0 T:10 20092 XUs and 13984 XVs -> 2645 XUVs
C:0 T:11 275 XUs and 193 XVs -> 23 XUVs
28641 XUVs total
8760 collection wire objects
25469 potential space points
Neighbour search...
2553019 tests to find 1248794 neighbours
Iterating with no regularization...
Begin: 6.19004e+07
0 5.44458e+07
1 5.39764e+07
2 5.39076e+07
3 5.38849e+07
Now with regularization...
Begin: 4.57028e+07
0 4.56846e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 411838253
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 411838253
fLifetime: 30000 [us]
19-Jan-2026 20:17:16 CET  Closed output file "reco_stage1.root"
19-Jan-2026 20:17:16 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          357.728       580.735       815.478       587.543       129.381        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00203574    0.00550893    0.00828195    0.00627173    0.00197901       10     
reco:ophit:OpHitFinder                             0.333011      0.432138      0.543141      0.433226      0.0657879       10     
reco:opflash:OpFlashFinder                         0.213013      0.273672      0.365741      0.263735      0.0480757       10     
reco:opslicer:OpSlicer                             0.750121       1.10414       1.60521       1.09399      0.237595        10     
reco:caldata:DataPrepModule                         5.50771       7.33657       9.12227       7.69777       1.17336        10     
reco:wclsdatasp:WireCellToolkit                     42.1791       47.4585       51.8008       47.024        2.89805        10     
reco:gaushit:GausHitFinder                          0.88584       1.92957        2.762        1.91853      0.560713        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000611707   0.000927036   0.00163871    0.000970202   0.000301459      10     
reco:reco3d:SpacePointSolver                        14.4456       23.9305       36.6659       23.0534       7.99203        10     
reco:hitpdune:DisambigFromSpacePoints               1.58523       2.62041       4.11075       2.27377      0.875096        10     
reco:pandora:StandardPandora                        24.8677       53.3647       79.9896       58.7903       16.9583        10     
reco:pandoraWriter:StandardPandora                 0.507777      0.744162      0.940747      0.754132      0.151849        10     
reco:pandoraTrack:LArPandoraTrackCreation           3.28231       7.31171       9.67547       7.56436       2.01385        10     
reco:pandoraShower:LArPandoraShowerCreation         2.73905       8.38404       13.3854       8.4471        3.31074        10     
reco:pandoracalo:Calorimetry                        1.78562       3.80924       5.55789       4.42005       1.31502        10     
reco:pandoracalonosce:Calorimetry                   1.58388       3.50595       5.21694       3.81696       1.28589        10     
reco:pandorapid:Chi2ParticleID                    0.00580955    0.00950893     0.0154422    0.00903996    0.00282554       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0518894     0.0903514     0.128013      0.0902534     0.0252269       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00550906    0.00863876     0.011752     0.00852927    0.00194481       10     
reco:pandoraShowercalo:ShowerCalorimetry            2.8223        7.72433       12.2404       7.37914       3.2648         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.70165       7.90315       14.0374       7.89792       3.24954        10     
reco:emtrkmichelid:EmTrackMichelId                  209.416       330.012       467.624       316.371       70.7255        10     
reco:crttag:SingleCRTMatchingProducer              0.0115435     0.0213874     0.0337921     0.0198228    0.00737996       10     
reco:crtreco:TwoCRTMatchingProducer               0.00693774     0.016547      0.0293571     0.0156182    0.00689607       10     
reco:anodepiercerst0:T0RecoAnodePiercers            0.90617       1.32736       1.84554       1.33358      0.285392        10     
reco:pandora2Track:LArPandoraTrackCreation          10.6325       22.3848       37.3335       22.8683       7.57598        10     
reco:pandora2calo:Calorimetry                       5.86995       10.6989       16.3334       10.8945       3.52527        10     
reco:pandora2calonosce:Calorimetry                  4.47783       9.6196        14.2551       10.6695       3.09813        10     
reco:pandora2pid:Chi2ParticleID                    0.0133438     0.0230405     0.0311065     0.0237487    0.00537647       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0588146     0.114588      0.168429      0.116395      0.0308491       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID          0.0132248     0.0220819     0.0307534     0.0214096    0.00528808       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.47436       7.48109       13.4347       7.38307       2.68124        10     
reco:pandora2Showercalo:ShowerCalorimetry           4.09382       8.33611       14.4035       8.15897       2.79295        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      3.85496       7.83284       12.9385       7.88743       2.60153        10     
[art]:TriggerResults:TriggerResultInserter        5.3651e-05    9.5562e-05    0.000240942   7.5406e-05    5.63446e-05      10     
end_path:out1:RootOutput                           1.218e-05    1.66042e-05    2.572e-05    1.52755e-05   4.68377e-06      10     
end_path:out1:RootOutput(write)                     2.84592       4.71544       6.42973       4.67478      0.975552        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5121.98 MB
  Peak resident set size usage (VmHWM): 4278.58 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.12 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.32 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   4.52 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.18 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.07 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10686052
-rw-r--r--. 1 fermilab users 2485655660 Jan 19 18:36 detsim_stage2.root
-rw-r--r--. 1 fermilab users 2355140915 Jan 19 18:32 detsim_stage1.root
-rw-r--r--. 1 fermilab users 2196320394 Jan 19 20:17 reco_stage1.root
-rw-r--r--. 1 fermilab users 1953398625 Jan 19 15:48 g4_stage2.root
-rw-r--r--. 1 fermilab users 1870929172 Jan 19 15:28 g4_stage1.root
-rw-r--r--. 1 fermilab users   80443130 Jan 19 20:11 Pandora_Events.pndr
-rw-r--r--. 1 fermilab users     242987 Jan 19 20:17 jobscript.log
-rw-r--r--. 1 fermilab users     217088 Jan 19 20:17 mem.db
-rw-r--r--. 1 fermilab users      62376 Jan 19 20:17 Pandora_Geometry.xml
-rw-r--r--. 1 fermilab users      36864 Jan 19 20:17 time.db
-rw-r--r--. 1 fermilab users        519 Jan 19 18:36 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 fermilab users        519 Jan 19 15:48 g4_protoDUNE_hist.root
-rw-r--r--. 1 fermilab users        519 Jan 19 20:17 protoDUNE_reco1_hist.root
-rw-r--r--. 1 fermilab users         52 Jan 19 15:20 all-input-dids.txt
-rw-r--r--. 1 fermilab users          7 Jan 19 20:17 justin-processed-pfns.txt
-rw-r--r--. 1 fermilab users          0 Jan 19 18:49 ProtoDUNE.log
-rw-r--r--. 1 fermilab users          0 Jan 19 18:36 debugprod.log
justIN time: 2026-02-05 17:07:59 UTC       justIN version: 01.06.00