justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 233778.0@dunegpschedd02.fnal.gov

Jobsub ID233778.0@dunegpschedd02.fnal.gov
Workflow ID9418
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-10-28 15:34:20
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2025-10-28 19:28:40
From worker nodeHostnamewn-lb-11.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-28 15:36:10
Input filesmonte-carlo-009418-000001
JobscriptExit code0
Real time3h (13846s)
CPU time3h (13693s = 98%)
Max RSS bytes4455837696 (4249 MiB)
Outputting started2025-10-28 19:26:57
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09418/1/001/reco_stage1.root
Finished2025-10-28 19:28:40
Saved logsjustin-logs:233778.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

dcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 126039 traces, 0 summary
Retagger: tagging trace set: gauss with 100178 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 100178 traces tagged "gauss"
FrameSaver: q=3.53982e+06 n=1831640 tag=gauss
wclsFrameSaver: saving 126039 traces tagged "wiener"
FrameSaver: q=3.73708e+06 n=1653050 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 33623 XUs and 37170 XVs -> 817 XUVs
C:0 T:1 76895 XUs and 89364 XVs -> 9606 XUVs
C:0 T:2 97170 XUs and 96217 XVs -> 5382 XUVs
C:0 T:3 42995 XUs and 39761 XVs -> 239 XUVs
C:0 T:4 14594 XUs and 17871 XVs -> 301 XUVs
C:0 T:5 80852 XUs and 114481 XVs -> 26202 XUVs
C:0 T:6 88656 XUs and 91175 XVs -> 2749 XUVs
C:0 T:7 35920 XUs and 35479 XVs -> 237 XUVs
C:0 T:8 11257 XUs and 8081 XVs -> 102 XUVs
C:0 T:9 75866 XUs and 62161 XVs -> 5123 XUVs
C:0 T:10 37668 XUs and 33638 XVs -> 3698 XUVs
C:0 T:11 273 XUs and 178 XVs -> 22 XUVs
54478 XUVs total
11004 collection wire objects
51329 potential space points
Neighbour search...
13587505 tests to find 5352824 neighbours
Iterating with no regularization...
Begin: 8.12444e+07
0 6.93009e+07
1 6.85811e+07
2 6.84921e+07
3 6.84648e+07
Now with regularization...
Begin: 5.87344e+07
0 5.8709e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410169587
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410169587
fLifetime: 30000 [us]
28-Oct-2025 20:26:55 CET  Closed output file "reco_stage1.root"
28-Oct-2025 20:26:55 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          261.581       420.514       648.993       369.84         115.5         10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.00120777    0.00558787    0.00884578     0.0060001    0.00242973       10     
reco:ophit:OpHitFinder                             0.284067      0.378639      0.496274      0.345052      0.0766746       10     
reco:opflash:OpFlashFinder                         0.167135      0.270087      0.410675      0.268499      0.0809377       10     
reco:opslicer:OpSlicer                             0.501182      0.739986       1.02582      0.677281      0.175231        10     
reco:caldata:DataPrepModule                         4.83333       5.6796        6.7949        5.57963      0.807019        10     
reco:wclsdatasp:WireCellToolkit                     36.2696       50.023        65.8302       49.4286       7.37673        10     
reco:gaushit:GausHitFinder                         0.933491       1.27167       1.71382       1.22429      0.213166        10     
reco:nhitsfilter:NumberOfHitsFilter               0.000352541   0.000735465   0.00119141    0.000688614   0.00028159       10     
reco:reco3d:SpacePointSolver                        7.73046       17.0948       30.579        14.8772       6.63457        10     
reco:hitpdune:DisambigFromSpacePoints              0.703486       1.64072       2.46054       1.49192      0.540528        10     
reco:pandora:StandardPandora                        14.2335       44.2415       90.8243       34.2715       23.072         10     
reco:pandoraWriter:StandardPandora                 0.319271      0.484721       0.6978       0.431312      0.119605        10     
reco:pandoraTrack:LArPandoraTrackCreation           3.03527       8.22864       14.4031       7.36556       3.71974        10     
reco:pandoraShower:LArPandoraShowerCreation         2.44646       9.96264       19.014        9.00607       5.02494        10     
reco:pandoracalo:Calorimetry                        1.71128       4.29182       7.43166       3.59486       1.90568        10     
reco:pandoracalonosce:Calorimetry                   1.40567       4.13661       7.49955       3.75949       1.82228        10     
reco:pandorapid:Chi2ParticleID                    0.00436441     0.0059722    0.00846901    0.00586633    0.00131245       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0353452     0.0499027     0.0777021     0.0467164     0.0119227       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00427988    0.00573074    0.00844584    0.00542619    0.00128261       10     
reco:pandoraShowercalo:ShowerCalorimetry            2.49912       9.63946       20.8242       8.65409       4.90909        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.1871        8.83898       16.521        8.09209       3.98743        10     
reco:emtrkmichelid:EmTrackMichelId                  124.499       177.567       253.23        164.268       37.2829        10     
reco:crttag:SingleCRTMatchingProducer              0.0135077     0.0218684     0.0332188     0.0210664    0.00663112       10     
reco:crtreco:TwoCRTMatchingProducer               0.00751318     0.0156223     0.0277817     0.0156524    0.00508575       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.571378       0.94947       1.59157       0.91203      0.291571        10     
reco:pandora2Track:LArPandoraTrackCreation          7.5897        24.744        47.5208       22.3955       11.2453        10     
reco:pandora2calo:Calorimetry                       3.8044        12.0118       22.2775       11.1534       5.19558        10     
reco:pandora2calonosce:Calorimetry                  3.42898       11.195        23.7934       9.8417        5.50312        10     
reco:pandora2pid:Chi2ParticleID                    0.0100346     0.0135388     0.0212757     0.0122449    0.00348686       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0439932     0.0651103     0.101928      0.0576116     0.0165184       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00957689     0.0136308     0.0208895     0.0122056    0.00349728       10     
reco:pandora2Shower:LArPandoraShowerCreation        2.92733       7.85745       16.2941       6.53518       3.71359        10     
reco:pandora2Showercalo:ShowerCalorimetry           3.22642       8.66383       16.3076       8.20993       3.55517        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.77489       7.01171       13.3992       6.96294       2.76871        10     
[art]:TriggerResults:TriggerResultInserter        3.0907e-05    5.27361e-05   0.000129013   4.55635e-05   2.59245e-05      10     
end_path:out1:RootOutput                           6.901e-06    8.5991e-06    1.4462e-05    7.8815e-06    2.08661e-06      10     
end_path:out1:RootOutput(write)                     2.09904       3.14741       4.82379       2.7985       0.771736        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5257.79 MB
  Peak resident set size usage (VmHWM): 4455.84 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.26 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.50 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.14 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.04 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10772184
-rw-r--r--. 1 dune009 dune 2520512412 Oct 28 19:14 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2392142230 Oct 28 19:12 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2230183640 Oct 28 20:26 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1945669289 Oct 28 16:58 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1860992096 Oct 28 16:43 g4_stage1.root
-rw-r--r--. 1 dune009 dune   80536080 Oct 28 20:22 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     242840 Oct 28 20:26 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Oct 28 20:26 mem.db
-rw-r--r--. 1 dune009 dune      62376 Oct 28 20:26 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Oct 28 20:26 time.db
-rw-r--r--. 1 dune009 dune        519 Oct 28 19:14 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Oct 28 16:58 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Oct 28 20:26 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Oct 28 16:36 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Oct 28 20:26 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Oct 28 19:20 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Oct 28 19:14 debugprod.log
justIN time: 2025-11-03 18:41:10 UTC       justIN version: 01.05.01