justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.55@dunegpschedd02.fnal.gov

Jobsub ID297921.55@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod03
Last heartbeat2026-02-04 01:00:29
From worker nodeHostnamewk03.grid.hep.ph.ic.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:49:00
Input filesvd-protodune:np02vd_raw_run042433_0539_df-s02-d1_dw_0_20260203T220657.hdf5
JobscriptExit code0
Real time1h (4145s)
CPU time1h (3778s = 91%)
Max RSS bytes3418107904 (3259 MiB)
Outputting started2026-02-04 00:58:08
Output files
Finished2026-02-04 01:00:29
Saved logsjustin-logs:297921.55-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ce point ptr... 0x19b2eb80, 0x19b2e6e0
Caught exception in looking up space point ptr... 0x595c24d0, 0x18341150
Caught exception in looking up space point ptr... 0x6295c090, 0x595c24d0
Caught exception in looking up space point ptr... 0x595a91e0, 0x5958e520
Caught exception in looking up space point ptr... 0x5958e520, 0x595a91e0
Caught exception in looking up space point ptr... 0x2f81e5d0, 0x2f81e100
Caught exception in looking up space point ptr... 0x2f81e100, 0x2f815100
Caught exception in looking up space point ptr... 0x647c8660, 0x647dab00
Caught exception in looking up space point ptr... 0x647dab00, 0x647da920
Caught exception in looking up space point ptr... 0x7558da10, 0x7038cda0
Caught exception in looking up space point ptr... 0x7038cda0, 0x70379250
Caught exception in looking up space point ptr... 0x17152c10, 0x17152da0
Caught exception in looking up space point ptr... 0x17152da0, 0x171530c0
Caught exception in looking up space point ptr... 0x5d8cd250, 0x5d8d2ca0
Caught exception in looking up space point ptr... 0x5d8d2ca0, 0x5d8cd250
Caught exception in looking up space point ptr... 0x12177a70, 0x82d2cf0
Caught exception in looking up space point ptr... 0x82d2cf0, 0x12177a70
Caught exception in looking up space point ptr... 0x2d34ff70, 0x2d351a50
Caught exception in looking up space point ptr... 0x2d351a50, 0x2d344450
Caught exception in looking up space point ptr... 0x2d3946c0, 0x2d394280
Caught exception in looking up space point ptr... 0x2d394280, 0x2d3946c0
Caught exception in looking up space point ptr... 0x2d3946c0, 0x2d394080
Caught exception in looking up space point ptr... 0x2d394080, 0x2d3946c0
Caught exception in looking up space point ptr... 0x187dc210, 0x187dc310
Caught exception in looking up space point ptr... 0x187dc310, 0x187dc210
Caught exception in looking up space point ptr... 0x54c1b830, 0x54c1d050
Caught exception in looking up space point ptr... 0x54c1d050, 0x54c18cb0
Caught exception in looking up space point ptr... 0xec18840, 0xe2291c0
Caught exception in looking up space point ptr... 0xe2291c0, 0xec186b0
Caught exception in looking up space point ptr... 0x15857190, 0x84116360
Caught exception in looking up space point ptr... 0x84116360, 0x15857190
Caught exception in looking up space point ptr... 0x7a567570, 0x648d0f20
Caught exception in looking up space point ptr... 0x648d0f20, 0x7a567570
Caught exception in looking up space point ptr... 0x42878fc0, 0x188b4fc0
Caught exception in looking up space point ptr... 0x188b4fc0, 0x42878fc0
Caught exception in looking up space point ptr... 0x3254dd10, 0x39699fd0
Caught exception in looking up space point ptr... 0x39699fd0, 0x3254dd10
Caught exception in looking up space point ptr... 0x3254dd10, 0x16ffbea0
Caught exception in looking up space point ptr... 0x16ffbea0, 0x3254dd10
Caught exception in looking up space point ptr... 0x11f00ad0, 0x11eff3d0
Caught exception in looking up space point ptr... 0x11eff3d0, 0x11f00ad0
Caught exception in looking up space point ptr... 0xe2570c0, 0xe2584d0
Caught exception in looking up space point ptr... 0xe2584d0, 0xe2570c0
Caught exception in looking up space point ptr... 0x2d34c350, 0x2d35df50
Caught exception in looking up space point ptr... 0x2d35df50, 0x2d34c350
Caught exception in looking up space point ptr... 0x62961fd0, 0x10685170
Caught exception in looking up space point ptr... 0x10685170, 0x658d31c0
Caught exception in looking up space point ptr... 0x2bd483c0, 0x2bd48230
Caught exception in looking up space point ptr... 0x2bd48230, 0x2bd48550
Caught exception in looking up space point ptr... 0x2f7f1340, 0x2f7f1210
Caught exception in looking up space point ptr... 0x2f7f1210, 0x2f7f1340
Caught exception in looking up space point ptr... 0x5113a160, 0x51137300
Caught exception in looking up space point ptr... 0x51137300, 0x511374c0
Caught exception in looking up space point ptr... 0x4e8c1df0, 0x46d72b80
Caught exception in looking up space point ptr... 0x46d72b80, 0x2c6f5d70
Caught exception in looking up space point ptr... 0x15a83160, 0x15a82930
Caught exception in looking up space point ptr... 0x15a82930, 0x15a82380
Caught exception in looking up space point ptr... 0xe4b7450, 0xe4b7360
Caught exception in looking up space point ptr... 0xe4b7360, 0xe4b7540
Caught exception in looking up space point ptr... 0x4b3eb870, 0xd4caf90
Caught exception in looking up space point ptr... 0xd4caf90, 0x4b3eb870
Caught exception in looking up space point ptr... 0x325003f0, 0x324ffea0
Caught exception in looking up space point ptr... 0x324ffea0, 0x325003f0
Caught exception in looking up space point ptr... 0x428d9ee0, 0x428d9bc0
Caught exception in looking up space point ptr... 0x428d9bc0, 0x428d9ee0
Caught exception in looking up space point ptr... 0x428c8a10, 0x428c80c0
Caught exception in looking up space point ptr... 0x428c80c0, 0x428c6d10
++++>>>> total num hits: 147456, num free: 142431
04-Feb-2026 00:57:56 GMT  Closed output file "np02vd_raw_run042433_0539_df-s02-d1_dw_0_20260203T220657_reco_stage0_20260204T005756_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         191.124       213.885       245.195       211.33        15.0589        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.8858e-05    7.97893e-05   0.000205254   6.6387e-05    3.30596e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                14.579        16.6098       19.6949        16.34        1.54174        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.504405       0.60368       1.0918       0.550479      0.129297        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0162322     0.0191522     0.0347172     0.0163505    0.00486364       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0161541     0.0163053     0.0165119     0.0162668    0.000104818      19     
produce:beamevent:BeamEvent                      7.5627e-05    0.000142816   0.000762278   0.000102659   0.000147257      19     
produce:pdvddaphne:DAPHNEReaderPDVD                5.30925       6.86859       10.1714       6.30465       1.2743         19     
produce:ophit:OpHitFinder                         0.0343823     0.0466614     0.0775272     0.0407182     0.0145811       19     
produce:wclsdatavd:WireCellToolkit                 142.85        149.449       155.751       148.832       3.54921        19     
produce:gaushit:GausHitFinder                      1.24574       1.91563       2.64319       1.90014      0.420427        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000377654   0.000600313   0.00103212    0.000564927   0.000155116      19     
produce:reco3d:SpacePointSolver                    12.7731       23.6047       44.1704       21.0178       8.66455        19     
produce:hitpdune:DisambigFromSpacePoints          0.153885      0.272493      0.557838      0.251519      0.0945549       19     
produce:cluster3d:Cluster3D                        1.87397       4.18026       10.5467       3.51555       2.23324        19     
[art]:TriggerResults:TriggerResultInserter       2.2643e-05    3.15884e-05   7.7692e-05    2.8575e-05    1.21231e-05      19     
end_path:out1:RootOutput                          3.324e-06    8.01437e-06   2.1806e-05     8.021e-06    4.15019e-06      19     
end_path:out1:RootOutput(write)                    8.70517       10.2622       14.1682       9.96588       1.19191        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5360.59 MB
  Peak resident set size usage (VmHWM): 3418.11 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0539_df-s02-d1_dw_0_20260203T220657_reco_stage0_20260204T005756_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0539_df-s02-d1_dw_0_20260203T220657_reco_stage0_20260204T005756_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0539_df-s02-d1_dw_0_20260203T220657_reco_stage0_20260204T005756_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162550.0,
    "core.end_time": 1770166680.0,
    "core.events": [
      61453,
      61459,
      61465,
      61471,
      61477,
      61483,
      61489,
      61495,
      61501,
      61507,
      61513,
      61519,
      61525,
      61531,
      61537,
      61543,
      61549,
      61555,
      61561
    ],
    "core.event_count": 19,
    "core.first_event_number": 61453,
    "core.last_event_number": 61561,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0539_df-s02-d1_dw_0_20260203T220657.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:28 UTC       justIN version: 01.06.00