justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303646.39@dunegpschedd01.fnal.gov

Jobsub ID303646.39@dunegpschedd01.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:44:24
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod01
Last heartbeat2026-02-04 00:26:46
From worker nodeHostnamebf1770077716.novalocal
cpuinfoAMD EPYC-Genoa Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:46:02
Input filesvd-protodune:np02vd_raw_run042422_0081_df-s03-d0_dw_0_20260203T155834.hdf5
JobscriptExit code0
Real time1h (5797s)
CPU time43m (2604s = 44%)
Max RSS bytes2777710592 (2649 MiB)
Outputting started2026-02-04 00:22:41
Output files
Finished2026-02-04 00:26:46
Saved logsjustin-logs:303646.39-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Caught exception in looking up space point ptr... 0x21fbb860, 0x63245c30
Caught exception in looking up space point ptr... 0x21fbbb80, 0x8272b50
Caught exception in looking up space point ptr... 0x8272b50, 0x21fbb860
Caught exception in looking up space point ptr... 0x4b766bb0, 0x4b75fe30
Caught exception in looking up space point ptr... 0x4b75f650, 0x4b766bb0
Caught exception in looking up space point ptr... 0x261b29a0, 0x4820bbc0
Caught exception in looking up space point ptr... 0x48321df0, 0x261b29a0
Caught exception in looking up space point ptr... 0x45144d50, 0x409a22a0
Caught exception in looking up space point ptr... 0x409a22a0, 0x63f8e520
Caught exception in looking up space point ptr... 0x263d47e0, 0x2e664be0
Caught exception in looking up space point ptr... 0x2e664be0, 0x272ea220
Caught exception in looking up space point ptr... 0x5e009450, 0x5e0098c0
Caught exception in looking up space point ptr... 0x5e0098c0, 0x5e009450
Caught exception in looking up space point ptr... 0x5e009450, 0x5e0095a0
Caught exception in looking up space point ptr... 0x5e0095a0, 0x5e009450
Caught exception in looking up space point ptr... 0x14fe3230, 0x2e766810
Caught exception in looking up space point ptr... 0x2e766810, 0x28d3a140
Caught exception in looking up space point ptr... 0x2eee35b0, 0x2eee32d0
Caught exception in looking up space point ptr... 0x2eee32d0, 0x2eee35b0
Caught exception in looking up space point ptr... 0xe212af0, 0xe212d50
Caught exception in looking up space point ptr... 0xe212d50, 0xe212e80
Caught exception in looking up space point ptr... 0x51fa6b10, 0x51fa6980
Caught exception in looking up space point ptr... 0x51fa6980, 0x51fa67f0
Caught exception in looking up space point ptr... 0x3d5ce800, 0x2e0f4e70
Caught exception in looking up space point ptr... 0x6049d790, 0x3d5ce800
Caught exception in looking up space point ptr... 0x4770ff50, 0x26d77d70
Caught exception in looking up space point ptr... 0x26d77d70, 0x394b3c30
Caught exception in looking up space point ptr... 0x22973560, 0x4770ff50
Caught exception in looking up space point ptr... 0x2b30ef00, 0x26f32b20
Caught exception in looking up space point ptr... 0x111b7640, 0x2b30ef00
Caught exception in looking up space point ptr... 0x268899b0, 0x15333840
Caught exception in looking up space point ptr... 0x15333840, 0x47699fa0
Caught exception in looking up space point ptr... 0x67e248a0, 0x67e236b0
Caught exception in looking up space point ptr... 0x67e236b0, 0x304cd0d0
Caught exception in looking up space point ptr... 0x5bd5acf0, 0xa0fcd80
Caught exception in looking up space point ptr... 0xa0fcd80, 0x3cfb81a0
Caught exception in looking up space point ptr... 0x2b0f1540, 0x38e5aee0
Caught exception in looking up space point ptr... 0x38e5a100, 0x2b0f1710
Caught exception in looking up space point ptr... 0x2b0f1710, 0x2b0f1540
Caught exception in looking up space point ptr... 0x14e56270, 0x539624e0
Caught exception in looking up space point ptr... 0x539624e0, 0x14e56270
Caught exception in looking up space point ptr... 0x63ba74c0, 0x4546ea20
Caught exception in looking up space point ptr... 0x4546ea20, 0x5cb16190
Caught exception in looking up space point ptr... 0x14eb0c60, 0x24696750
Caught exception in looking up space point ptr... 0x24696750, 0x14eb0c60
Caught exception in looking up space point ptr... 0x3e8b0330, 0x154d6fe0
Caught exception in looking up space point ptr... 0x154d6fe0, 0x3e8b0330
Caught exception in looking up space point ptr... 0x3aace8a0, 0x134cf0c0
Caught exception in looking up space point ptr... 0x134cf0c0, 0x3aace8a0
Caught exception in looking up space point ptr... 0x3aace8a0, 0x217024c0
Caught exception in looking up space point ptr... 0x217024c0, 0x3aace8a0
Caught exception in looking up space point ptr... 0x45a6b480, 0x202d3780
Caught exception in looking up space point ptr... 0x202d3780, 0x44a4ba90
Caught exception in looking up space point ptr... 0x2357dc90, 0x35fc27a0
Caught exception in looking up space point ptr... 0x35fc27a0, 0x2357db60
Caught exception in looking up space point ptr... 0xaa55a20, 0x35fbde60
Caught exception in looking up space point ptr... 0x35fbde60, 0x63b9b450
Caught exception in looking up space point ptr... 0x4faf9c70, 0x2bf352f0
Caught exception in looking up space point ptr... 0x2bf352f0, 0x4faf9c70
Caught exception in looking up space point ptr... 0x1407bc90, 0x35fc3100
Caught exception in looking up space point ptr... 0x35fc3100, 0x63b942c0
Caught exception in looking up space point ptr... 0x39a29320, 0x2ff42aa0
Caught exception in looking up space point ptr... 0x2ff42aa0, 0x482c8ae0
Caught exception in looking up space point ptr... 0x40f0faa0, 0x40f0fe10
Caught exception in looking up space point ptr... 0x40f0fe10, 0x40f0ffa0
++++>>>> total num hits: 240412, num free: 233640
04-Feb-2026 00:22:27 GMT  Closed output file "np02vd_raw_run042422_0081_df-s03-d0_dw_0_20260203T155834_reco_stage0_20260204T002227_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         155.496       211.486       283.859       202.433       30.8376        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.185e-05    0.000102243   0.000361289    7.858e-05    6.70995e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                65.6249       111.789       167.564       112.407       20.3348        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.592014      0.824725       1.59517      0.780521      0.221883        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0177025     0.0252193     0.0681456     0.0213641     0.0101382       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.086698      0.0995304     0.136863      0.0982746     0.0102245       27     
produce:beamevent:BeamEvent                       6.823e-05    0.000295151   0.00184287    0.000135629   0.000354682      27     
produce:pdvddaphne:DAPHNEReaderPDVD                7.3288        13.1748       19.4338       13.0851       2.83226        27     
produce:ophit:OpHitFinder                         0.0218989     0.0342171     0.0439236     0.0343313    0.00518178       27     
produce:wclsdatavd:WireCellToolkit                 47.7923       54.3766       63.3258       53.6899       3.76295        27     
produce:gaushit:GausHitFinder                     0.591931      0.987488       1.94477      0.959229      0.263874        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000149729   0.000276863   0.000939396   0.000219529   0.00016612       27     
produce:reco3d:SpacePointSolver                    5.79392       9.1489        20.3242       8.46214       2.95186        27     
produce:hitpdune:DisambigFromSpacePoints          0.0670059     0.147342      0.407088       0.1329       0.062678        27     
produce:cluster3d:Cluster3D                        1.24177       17.1935       60.1366       17.5401       11.2701        27     
[art]:TriggerResults:TriggerResultInserter        1.941e-05    3.0162e-05    0.000151949    2.182e-05    2.60815e-05      27     
end_path:out1:RootOutput                          3.47e-06     6.93441e-06    2.803e-05     5.25e-06     5.23178e-06      27     
end_path:out1:RootOutput(write)                    2.82988       3.63957       5.33506       3.59789      0.439641        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4676.71 MB
  Peak resident set size usage (VmHWM): 2777.71 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0081_df-s03-d0_dw_0_20260203T155834_reco_stage0_20260204T002227_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0081_df-s03-d0_dw_0_20260203T155834_reco_stage0_20260204T002227_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0081_df-s03-d0_dw_0_20260203T155834_reco_stage0_20260204T002227_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158776.0,
    "core.end_time": 1770164550.0,
    "core.events": [
      13136,
      13142,
      13148,
      13154,
      13160,
      13166,
      13172,
      13178,
      13184,
      13190,
      13196,
      13202,
      13208,
      13214,
      13220,
      13226,
      13232,
      13238,
      13244,
      13250,
      13256,
      13262,
      13268,
      13274,
      13280,
      13286,
      13292
    ],
    "core.event_count": 27,
    "core.first_event_number": 13136,
    "core.last_event_number": 13292,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0081_df-s03-d0_dw_0_20260203T155834.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:59:19 UTC       justIN version: 01.06.00