justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 239644.0@dunegpschedd01.fnal.gov

Jobsub ID239644.0@dunegpschedd01.fnal.gov
Workflow ID9426
Stage ID1
User namepmelas@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8388608000 (8000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2025-10-28 23:31:44
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc01
Last heartbeat2025-10-29 03:10:08
From worker nodeHostnamewn-la-16.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-28 23:33:20
Input filesmonte-carlo-009426-000001
JobscriptExit code0
Real time3h (12890s)
CPU time3h (12753s = 98%)
Max RSS bytes4517593088 (4308 MiB)
Outputting started2025-10-29 03:08:11
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/pmelas/fnal/09426/1/001/reco_stage1.root
Finished2025-10-29 03:10:08
Saved logsjustin-logs:239644.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

adcKeepAllSignalFinder
ToolBasedRawDigitPrepService:prepare:   Running tool adcScaleKeToAdc
ToolBasedRawDigitPrepService:prepare:   Running tool pdsp_RemoveBadChannels
ToolBasedRawDigitPrepService:endEvent: End processing run 1 event 10 with 7 tools.
CookedFrameSource: got 15360 recob::Wire objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 128339 traces, 0 summary
Retagger: tagging trace set: gauss with 99981 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 99981 traces tagged "gauss"
FrameSaver: q=3.52333e+06 n=1818213 tag=gauss
wclsFrameSaver: saving 128339 traces tagged "wiener"
FrameSaver: q=3.71654e+06 n=1631266 tag=wiener
72 X, 28 U, 34 V bad channels
Finding XUV coincidences...
C:0 T:0 32993 XUs and 37957 XVs -> 448 XUVs
C:0 T:1 59371 XUs and 62092 XVs -> 3814 XUVs
C:0 T:2 104861 XUs and 99325 XVs -> 10787 XUVs
C:0 T:3 39965 XUs and 23178 XVs -> 412 XUVs
C:0 T:4 15134 XUs and 13604 XVs -> 134 XUVs
C:0 T:5 52916 XUs and 38175 XVs -> 4143 XUVs
C:0 T:6 117992 XUs and 120759 XVs -> 5510 XUVs
C:0 T:7 45702 XUs and 46888 XVs -> 561 XUVs
C:0 T:8 6852 XUs and 6577 XVs -> 67 XUVs
C:0 T:9 43963 XUs and 52490 XVs -> 3317 XUVs
C:0 T:10 37488 XUs and 27818 XVs -> 4464 XUVs
C:0 T:11 649 XUs and 374 XVs -> 56 XUVs
33713 XUVs total
11911 collection wire objects
30486 potential space points
Neighbour search...
2896428 tests to find 1393536 neighbours
Iterating with no regularization...
Begin: 7.34175e+07
0 6.56748e+07
1 6.50909e+07
2 6.50225e+07
3 6.50047e+07
Now with regularization...
Begin: 5.48748e+07
0 5.48627e+07
run: 1 ; subrun: 0 ; event: 10
evttime: 410176345
fLifetime: 30000 [us]
run: 1 ; subrun: 0 ; event: 10
evttime: 410176345
fLifetime: 30000 [us]
29-Oct-2025 04:08:09 CET  Closed output file "reco_stage1.root"
29-Oct-2025 04:08:09 CET  Closed input file "detsim_stage2.root"
DataPrepModule::endJob: # events processed: 10
DataPrepModule::endJob:   # events skipped: 0

====================================================================================================================================
TimeTracker printout (sec)                            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================================
Full event                                          250.657       376.886       488.989       393.039       72.2542        10     
------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                            0.000867658   0.00529027    0.00859132    0.00569958    0.00246398       10     
reco:ophit:OpHitFinder                             0.306838      0.343637      0.423167       0.33376      0.034082        10     
reco:opflash:OpFlashFinder                         0.208134      0.243615      0.313317      0.236251      0.029792        10     
reco:opslicer:OpSlicer                             0.601706      0.681326      0.769336       0.68699      0.0532881       10     
reco:caldata:DataPrepModule                         4.76209       5.01384       5.43645       4.8742       0.243184        10     
reco:wclsdatasp:WireCellToolkit                      36.03        48.6973       55.6837       50.8762       6.52059        10     
reco:gaushit:GausHitFinder                         0.805374       1.15687       1.46146       1.18737      0.194348        10     
reco:nhitsfilter:NumberOfHitsFilter               0.00030712    0.000487452   0.000760308   0.00043115    0.00014283       10     
reco:reco3d:SpacePointSolver                        8.06945       16.0145       23.6031       16.4809       4.98732        10     
reco:hitpdune:DisambigFromSpacePoints              0.795857       1.50471       2.40635       1.50833      0.444885        10     
reco:pandora:StandardPandora                        16.7287       36.7499       74.3587       34.198        16.0133        10     
reco:pandoraWriter:StandardPandora                 0.301015      0.441247      0.630666      0.439176      0.0919852       10     
reco:pandoraTrack:LArPandoraTrackCreation           2.73235       6.09843       9.95907       6.40746       1.96969        10     
reco:pandoraShower:LArPandoraShowerCreation         2.82327       7.32268       11.5852       7.52765       2.82425        10     
reco:pandoracalo:Calorimetry                        1.46597       3.32744       6.55772       3.30008       1.38909        10     
reco:pandoracalonosce:Calorimetry                   1.3867        3.12361       5.6839        3.1292        1.2077         10     
reco:pandorapid:Chi2ParticleID                    0.00394162    0.00613508     0.0139283    0.00546633    0.00273159       10     
reco:pandoracalinoxyzt:CalibrationdEdXPDSP         0.0334545      0.04601      0.0593432     0.0470444    0.00810153       10     
reco:pandoracalipidnoxyzt:Chi2ParticleID          0.00373155    0.00506644    0.00651311    0.00522835    0.000801536      10     
reco:pandoraShowercalo:ShowerCalorimetry            2.61508       7.37012       12.4641       7.60261       2.96702        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       2.50084       7.01858       11.175        7.32913       2.68321        10     
reco:emtrkmichelid:EmTrackMichelId                  121.27        171.493       214.438       176.46        29.0517        10     
reco:crttag:SingleCRTMatchingProducer             0.00839863     0.0177191     0.0323685     0.0158268    0.00713406       10     
reco:crtreco:TwoCRTMatchingProducer               0.00739185     0.0118476     0.0148249     0.0125032    0.00257604       10     
reco:anodepiercerst0:T0RecoAnodePiercers           0.601871      0.876007       1.22638      0.857437      0.178208        10     
reco:pandora2Track:LArPandoraTrackCreation          8.61654       18.9087       28.6834       21.1174        6.293         10     
reco:pandora2calo:Calorimetry                       3.98287       9.13121       12.9463       9.74758       2.94033        10     
reco:pandora2calonosce:Calorimetry                  3.77195       8.67088       11.807        9.23762       2.62525        10     
reco:pandora2pid:Chi2ParticleID                   0.00867259     0.012622      0.015355      0.0128675    0.00226266       10     
reco:pandora2calinoxyzt:CalibrationdEdXPDSP        0.0425508     0.0593089     0.0741524     0.0602326     0.0102283       10     
reco:pandora2calipidnoxyzt:Chi2ParticleID         0.00849997     0.0125957     0.0149447     0.0126197    0.00213579       10     
reco:pandora2Shower:LArPandoraShowerCreation        3.14856       6.25302       8.61757       6.70413       1.6921         10     
reco:pandora2Showercalo:ShowerCalorimetry           3.37722       7.22006       9.14845       7.82821       2.03437        10     
reco:pandora2Showercalonosce:ShowerCalorimetry      2.84741       5.97547       7.66742       6.26052       1.59063        10     
[art]:TriggerResults:TriggerResultInserter        3.1427e-05    4.73014e-05   0.000114952   3.7005e-05    2.46292e-05      10     
end_path:out1:RootOutput                           6.541e-06    7.7498e-06     1.26e-05      7.231e-06    1.66645e-06      10     
end_path:out1:RootOutput(write)                     2.03699       2.87526       3.64951       2.90799      0.515857        10     
====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5320.22 MB
  Peak resident set size usage (VmHWM): 4517.59 MB
  Details saved in: 'mem.db'
====================================================================================================
ToolBasedRawDigitPrepService:dtor: Event count: 10
ToolBasedRawDigitPrepService:dtor:  Call count: 60
ToolBasedRawDigitPrepService:dtor: Time report for 7 tools.
ToolBasedRawDigitPrepService:dtor:                    digitReader:   0.06 sec/event
ToolBasedRawDigitPrepService:dtor:                adcSampleFiller:   0.22 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleAdcToKe:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:          pdspTailPedRemovalZKe:   3.09 sec/event
ToolBasedRawDigitPrepService:dtor:         adcKeepAllSignalFinder:   0.13 sec/event
ToolBasedRawDigitPrepService:dtor:                adcScaleKeToAdc:   0.03 sec/event
ToolBasedRawDigitPrepService:dtor:         pdsp_RemoveBadChannels:   0.00 sec/event
Art has completed and will exit with status 0.
lar exit code 0
.:
total 10802384
-rw-r--r--. 1 dune009 dune 2514313314 Oct 29 03:03 detsim_stage2.root
-rw-r--r--. 1 dune009 dune 2382359293 Oct 29 03:01 detsim_stage1.root
-rw-r--r--. 1 dune009 dune 2227110803 Oct 29 04:08 reco_stage1.root
-rw-r--r--. 1 dune009 dune 1964944992 Oct 29 00:51 g4_stage2.root
-rw-r--r--. 1 dune009 dune 1890735842 Oct 29 00:39 g4_stage1.root
-rw-r--r--. 1 dune009 dune   81562600 Oct 29 04:02 Pandora_Events.pndr
-rw-r--r--. 1 dune009 dune     243092 Oct 29 04:08 jobscript.log
-rw-r--r--. 1 dune009 dune     217088 Oct 29 04:08 mem.db
-rw-r--r--. 1 dune009 dune      62376 Oct 29 04:08 Pandora_Geometry.xml
-rw-r--r--. 1 dune009 dune      36864 Oct 29 04:08 time.db
-rw-r--r--. 1 dune009 dune        519 Oct 29 03:03 detsim_single_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Oct 29 00:51 g4_protoDUNE_hist.root
-rw-r--r--. 1 dune009 dune        519 Oct 29 04:08 protoDUNE_reco1_hist.root
-rw-r--r--. 1 dune009 dune         52 Oct 29 00:33 all-input-dids.txt
-rw-r--r--. 1 dune009 dune          7 Oct 29 04:08 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune          0 Oct 29 03:09 ProtoDUNE.log
-rw-r--r--. 1 dune009 dune          0 Oct 29 03:03 debugprod.log
justIN time: 2025-11-03 18:40:58 UTC       justIN version: 01.05.01