justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303649.0@dunegpschedd01.fnal.gov

Jobsub ID303649.0@dunegpschedd01.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:48:26
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod00
Last heartbeat2026-02-04 00:33:14
From worker nodeHostnamebf1770081313.novalocal
cpuinfoAMD EPYC-Rome Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:53:47
Input filesvd-protodune:np02vd_raw_run042422_0113_df-s04-d1_dw_0_20260203T161552.hdf5
JobscriptExit code0
Real time1h (5758s)
CPU time37m (2236s = 38%)
Max RSS bytes2591735808 (2471 MiB)
Outputting started2026-02-04 00:29:46
Output files
Finished2026-02-04 00:33:14
Saved logsjustin-logs:303649.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

x66074bb0
Caught exception in looking up space point ptr... 0x66074bb0, 0x66074e00
Caught exception in looking up space point ptr... 0x15f8a4b0, 0x15f85bc0
Caught exception in looking up space point ptr... 0x15f85bc0, 0x15f8aa10
Caught exception in looking up space point ptr... 0x5e93c710, 0x5e9351b0
Caught exception in looking up space point ptr... 0x5e9351b0, 0x5e931840
Caught exception in looking up space point ptr... 0x352bc340, 0x5e93c710
Caught exception in looking up space point ptr... 0x123fa620, 0x123fa4d0
Caught exception in looking up space point ptr... 0x123fcc50, 0x123fa620
Caught exception in looking up space point ptr... 0x53b0f1b0, 0x53b14ad0
Caught exception in looking up space point ptr... 0x53b0ca90, 0x53b0f1b0
Caught exception in looking up space point ptr... 0x504e5e50, 0x21f53f20
Caught exception in looking up space point ptr... 0x21f53f20, 0x504e2260
Caught exception in looking up space point ptr... 0x12173220, 0x12173180
Caught exception in looking up space point ptr... 0x12173180, 0x12173220
Caught exception in looking up space point ptr... 0x499933c0, 0x4998ca90
Caught exception in looking up space point ptr... 0x4998ca90, 0x499933c0
Caught exception in looking up space point ptr... 0x570bb4a0, 0x570d0ea0
Caught exception in looking up space point ptr... 0x570c5f30, 0x570bb4a0
Caught exception in looking up space point ptr... 0x504812e0, 0x504891a0
Caught exception in looking up space point ptr... 0x50488080, 0x504812e0
Caught exception in looking up space point ptr... 0x5e930a60, 0x5e92afe0
Caught exception in looking up space point ptr... 0x5e92afe0, 0x5e930a60
Caught exception in looking up space point ptr... 0x14223f50, 0x45bc9a50
Caught exception in looking up space point ptr... 0x45bc9a50, 0x458ed210
Caught exception in looking up space point ptr... 0x2e167d20, 0x2e167b90
Caught exception in looking up space point ptr... 0x2e167b90, 0x2e167d20
Caught exception in looking up space point ptr... 0xef4aa30, 0xef4af80
Caught exception in looking up space point ptr... 0xef4af80, 0xef4aa30
Caught exception in looking up space point ptr... 0x4a7c3d60, 0x4d1b92a0
Caught exception in looking up space point ptr... 0x4d1b92a0, 0x4a7c3d60
Caught exception in looking up space point ptr... 0x3fafc1e0, 0x3fafc480
Caught exception in looking up space point ptr... 0x3fafc480, 0x3fafbf40
Caught exception in looking up space point ptr... 0xef61740, 0xef61a30
Caught exception in looking up space point ptr... 0xef61a30, 0xef61740
Caught exception in looking up space point ptr... 0x352c1890, 0x352c2be0
Caught exception in looking up space point ptr... 0x352c2be0, 0x352c1890
Caught exception in looking up space point ptr... 0x220a88b0, 0x220a69c0
Caught exception in looking up space point ptr... 0x220a69c0, 0x220a88b0
Caught exception in looking up space point ptr... 0x14234dc0, 0x2e159060
Caught exception in looking up space point ptr... 0x2e159060, 0x14234dc0
Caught exception in looking up space point ptr... 0x1126a430, 0x1126a5d0
Caught exception in looking up space point ptr... 0x1126a5d0, 0x1126a430
Caught exception in looking up space point ptr... 0x11253280, 0x11253350
Caught exception in looking up space point ptr... 0x11253350, 0x11253280
Caught exception in looking up space point ptr... 0x62aef430, 0x62aef070
Caught exception in looking up space point ptr... 0x62aef070, 0x62aef300
Caught exception in looking up space point ptr... 0x62aef300, 0x62af03f0
Caught exception in looking up space point ptr... 0x352c3450, 0x352c8560
Caught exception in looking up space point ptr... 0x352c8560, 0x352c3450
Caught exception in looking up space point ptr... 0x12b7fd40, 0x15c52e70
Caught exception in looking up space point ptr... 0x15c52e70, 0x12b7fd40
Caught exception in looking up space point ptr... 0x125a6170, 0x125a49b0
Caught exception in looking up space point ptr... 0x125a49b0, 0x125a4ac0
Caught exception in looking up space point ptr... 0x125a4ac0, 0x125a4f40
Caught exception in looking up space point ptr... 0x125a4f40, 0x125a6170
Caught exception in looking up space point ptr... 0x2e146910, 0x2e146e60
Caught exception in looking up space point ptr... 0x2e146e60, 0x2e146910
Caught exception in looking up space point ptr... 0x4c56b170, 0x4c56b490
Caught exception in looking up space point ptr... 0x4c56b490, 0x4c56b170
Caught exception in looking up space point ptr... 0x5cf198f0, 0x5cf2b540
Caught exception in looking up space point ptr... 0x5cf2b540, 0x5cf198f0
Caught exception in looking up space point ptr... 0xe770320, 0xe772f20
Caught exception in looking up space point ptr... 0xe772f20, 0xe770320
Caught exception in looking up space point ptr... 0xe75c4d0, 0xe75c2f0
Caught exception in looking up space point ptr... 0xe75c2f0, 0xe75c4d0
++++>>>> total num hits: 58105, num free: 55762
04-Feb-2026 00:29:36 GMT  Closed output file "np02vd_raw_run042422_0113_df-s04-d1_dw_0_20260203T161552_reco_stage0_20260204T002936_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         151.78        210.506       279.32        207.774       29.1048        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.4304e-05    6.20853e-05   0.00013882    5.4532e-05    1.9705e-05       27     
produce:tpcrawdecoder:PDVDTPCReader                69.5483       120.949       186.161       117.43        25.8778        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.576226      0.917645       3.29764      0.717271       0.62021        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0165905     0.0298786     0.262346      0.0175257     0.0462219       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0826324     0.087278      0.102414      0.0854174     0.0049414       27     
produce:beamevent:BeamEvent                      4.7308e-05    9.82671e-05   0.000850093   6.2487e-05    0.000149582      27     
produce:pdvddaphne:DAPHNEReaderPDVD                8.63306       14.5485       21.106        14.3095       3.04434        27     
produce:ophit:OpHitFinder                         0.0202601     0.032206      0.0375495     0.0332602     0.0037695       27     
produce:wclsdatavd:WireCellToolkit                 58.3081       61.3379       72.8042       59.9694       3.33286        27     
produce:gaushit:GausHitFinder                     0.476942      0.731575       1.20116      0.660592       0.17718        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000119674   0.000201651   0.000363991   0.000194906   6.26449e-05      27     
produce:reco3d:SpacePointSolver                    4.34098       7.06253       14.0834       6.44842       2.19778        27     
produce:hitpdune:DisambigFromSpacePoints          0.0522406     0.0809897     0.198553      0.0742653     0.0296911       27     
produce:cluster3d:Cluster3D                       0.527535       1.26769       4.29449       1.15883      0.736029        27     
[art]:TriggerResults:TriggerResultInserter       1.8104e-05    2.6146e-05    0.000117371   1.9938e-05    2.06647e-05      27     
end_path:out1:RootOutput                          3.136e-06    6.30959e-06   3.8663e-05     4.117e-06    6.96113e-06      27     
end_path:out1:RootOutput(write)                    3.20776       3.44457       4.13142       3.39174      0.196843        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4489.45 MB
  Peak resident set size usage (VmHWM): 2591.74 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0113_df-s04-d1_dw_0_20260203T161552_reco_stage0_20260204T002936_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0113_df-s04-d1_dw_0_20260203T161552_reco_stage0_20260204T002936_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0113_df-s04-d1_dw_0_20260203T161552_reco_stage0_20260204T002936_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770159235.0,
    "core.end_time": 1770164977.0,
    "core.events": [
      18323,
      18329,
      18335,
      18341,
      18347,
      18353,
      18359,
      18365,
      18371,
      18377,
      18383,
      18389,
      18395,
      18401,
      18407,
      18413,
      18419,
      18425,
      18431,
      18437,
      18443,
      18449,
      18455,
      18461,
      18467,
      18473,
      18479
    ],
    "core.event_count": 27,
    "core.first_event_number": 18323,
    "core.last_event_number": 18479,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0113_df-s04-d1_dw_0_20260203T161552.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:58:07 UTC       justIN version: 01.06.00