justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.138@dunegpschedd02.fnal.gov

Jobsub ID297921.138@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce02
Last heartbeat2026-02-04 02:00:42
From worker nodeHostnamecn046.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:56:21
Input filesvd-protodune:np02vd_raw_run042433_0615_df-s02-d0_dw_0_20260203T223551.hdf5
JobscriptExit code0
Real time2h (7291s)
CPU time1h (4051s = 55%)
Max RSS bytes3344224256 (3189 MiB)
Outputting started2026-02-04 01:57:57
Output files
Finished2026-02-04 02:00:42
Saved logsjustin-logs:297921.138-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

oint ptr... 0x3a5eae70, 0x3a5e8370
Caught exception in looking up space point ptr... 0x3a5e8370, 0x3a5e8470
Caught exception in looking up space point ptr... 0x3a5e8470, 0x3a5e8270
Caught exception in looking up space point ptr... 0x390631a0, 0x390652a0
Caught exception in looking up space point ptr... 0x39066ea0, 0x390631a0
Caught exception in looking up space point ptr... 0x431ca5c0, 0x431ca6a0
Caught exception in looking up space point ptr... 0x431ca6a0, 0x431ca5c0
Caught exception in looking up space point ptr... 0x7dfc5f90, 0x7dfc19c0
Caught exception in looking up space point ptr... 0x7dfc19c0, 0x18b07a90
Caught exception in looking up space point ptr... 0x41866120, 0x418662b0
Caught exception in looking up space point ptr... 0x418662b0, 0x41866120
Caught exception in looking up space point ptr... 0x6ca6dfe0, 0x6ca6e0e0
Caught exception in looking up space point ptr... 0x6ca6e0e0, 0x6ca6dfe0
Caught exception in looking up space point ptr... 0xe8de980, 0x65bdaf50
Caught exception in looking up space point ptr... 0x65bdaf50, 0xe8de980
Caught exception in looking up space point ptr... 0x5d832160, 0x3c4fd2e0
Caught exception in looking up space point ptr... 0x3c4fd2e0, 0x39c895f0
Caught exception in looking up space point ptr... 0x164b3e70, 0x164b22a0
Caught exception in looking up space point ptr... 0x164b22a0, 0x164b24f0
Caught exception in looking up space point ptr... 0x49b67b80, 0x3a602460
Caught exception in looking up space point ptr... 0x3a602460, 0x49b67b80
Caught exception in looking up space point ptr... 0x49b67b80, 0xbac2440
Caught exception in looking up space point ptr... 0xbac2440, 0x49b67b80
Caught exception in looking up space point ptr... 0x5a14edb0, 0x39fb0180
Caught exception in looking up space point ptr... 0x5a14b9b0, 0x5a14f9b0
Caught exception in looking up space point ptr... 0x5a14f9b0, 0x5a14dc50
Caught exception in looking up space point ptr... 0x5a14dc50, 0x5a14edb0
Caught exception in looking up space point ptr... 0x620aa5d0, 0x620ae000
Caught exception in looking up space point ptr... 0x620ae000, 0x620addb0
Caught exception in looking up space point ptr... 0x7a5065c0, 0x7a506750
Caught exception in looking up space point ptr... 0x7a506750, 0x7a5068e0
Caught exception in looking up space point ptr... 0x49be9350, 0x49be8ea0
Caught exception in looking up space point ptr... 0x49be8ea0, 0x49be9350
Caught exception in looking up space point ptr... 0x3b6b2a90, 0x46c7d6d0
Caught exception in looking up space point ptr... 0x46c7d6d0, 0x3b6b2a90
Caught exception in looking up space point ptr... 0xf386740, 0x2b6de7a0
Caught exception in looking up space point ptr... 0x2b6de7a0, 0xf386740
Caught exception in looking up space point ptr... 0x2e4ace00, 0x2e4ad0a0
Caught exception in looking up space point ptr... 0x2e4ad0a0, 0x2e4ace00
Caught exception in looking up space point ptr... 0x5a15b4b0, 0x5a15b3b0
Caught exception in looking up space point ptr... 0x5a15b3b0, 0x5a15b4b0
Caught exception in looking up space point ptr... 0x5a15b4b0, 0x5a15aec0
Caught exception in looking up space point ptr... 0x5a15aec0, 0x5a15b4b0
Caught exception in looking up space point ptr... 0x3a82af60, 0x3a827d10
Caught exception in looking up space point ptr... 0x3a827d10, 0x16c6fca0
Caught exception in looking up space point ptr... 0x7dc008c0, 0x7dc0ac80
Caught exception in looking up space point ptr... 0x7dc0ac80, 0x7dc04830
Caught exception in looking up space point ptr... 0x43a6de50, 0x43a6e170
Caught exception in looking up space point ptr... 0x43a6e170, 0x43a6e300
Caught exception in looking up space point ptr... 0xedfa480, 0xedfb640
Caught exception in looking up space point ptr... 0xedfb640, 0xedfa480
Caught exception in looking up space point ptr... 0x84921980, 0x84921cf0
Caught exception in looking up space point ptr... 0x84921cf0, 0x84921980
Caught exception in looking up space point ptr... 0x851c8df0, 0x851c8f80
Caught exception in looking up space point ptr... 0x851c8f80, 0x851c8df0
Caught exception in looking up space point ptr... 0x61c6ca20, 0x45694560
Caught exception in looking up space point ptr... 0x45694560, 0x45694800
Caught exception in looking up space point ptr... 0x6d8cbfd0, 0x6d8cde80
Caught exception in looking up space point ptr... 0x6d8cde80, 0x6d8cbfd0
Caught exception in looking up space point ptr... 0x7d396b50, 0x7d39d400
Caught exception in looking up space point ptr... 0x7d39d400, 0x7d39cee0
Caught exception in looking up space point ptr... 0x13a54660, 0x13a54540
Caught exception in looking up space point ptr... 0x13a54540, 0x13a54660
Caught exception in looking up space point ptr... 0x60eb0b40, 0x6f1d1a50
Caught exception in looking up space point ptr... 0x6f1d1a50, 0x60eb0b40
Caught exception in looking up space point ptr... 0x7dae6f30, 0x60b4def0
Caught exception in looking up space point ptr... 0x60b4def0, 0x7dae6f30
++++>>>> total num hits: 97415, num free: 93899
04-Feb-2026 01:57:42 GMT  Closed output file "np02vd_raw_run042433_0615_df-s02-d0_dw_0_20260203T223551_reco_stage0_20260204T015742_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         317.265       377.604       460.172       371.253       34.1376        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.5339e-05    0.000118377   0.00025538    0.000114346   3.32023e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                100.14        162.88        233.268       163.781       30.1161        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.575483      0.764081       1.19966      0.748532      0.154512        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.016603      0.0207638     0.028237      0.0204237    0.00348467       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0166298     0.0176948     0.0327017     0.016805     0.00354584       19     
produce:beamevent:BeamEvent                      0.000133487   0.000222994   0.00128744    0.000158534   0.000251785      19     
produce:pdvddaphne:DAPHNEReaderPDVD                14.7638       17.8582       22.1513       17.2346       2.08222        19     
produce:ophit:OpHitFinder                         0.0354393     0.0532573     0.0737557     0.0533988     0.0128907       19     
produce:wclsdatavd:WireCellToolkit                 137.602       170.641       225.47        156.222       28.7286        19     
produce:gaushit:GausHitFinder                      1.17255       1.67949       2.61375       1.53716      0.382568        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000504321   0.000613007   0.000797293   0.000592879   7.91697e-05      19     
produce:reco3d:SpacePointSolver                    10.1518       15.2237       22.5828       14.0277       3.0834         19     
produce:hitpdune:DisambigFromSpacePoints          0.143346      0.250837      0.405013      0.231305      0.0729347       19     
produce:cluster3d:Cluster3D                        1.26442       2.29107       3.83496       1.97229       0.81602        19     
[art]:TriggerResults:TriggerResultInserter       2.9874e-05    4.33577e-05   8.0695e-05    3.9212e-05    1.19729e-05      19     
end_path:out1:RootOutput                          4.024e-06    7.45621e-06   2.2581e-05     6.356e-06    4.1224e-06       19     
end_path:out1:RootOutput(write)                    5.00013       5.87445       7.90759       5.59736      0.877663        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5300.98 MB
  Peak resident set size usage (VmHWM): 3344.22 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0615_df-s02-d0_dw_0_20260203T223551_reco_stage0_20260204T015742_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0615_df-s02-d0_dw_0_20260203T223551_reco_stage0_20260204T015742_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0615_df-s02-d0_dw_0_20260203T223551_reco_stage0_20260204T015742_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162993.0,
    "core.end_time": 1770170265.0,
    "core.events": [
      70116,
      70122,
      70128,
      70134,
      70140,
      70146,
      70152,
      70158,
      70164,
      70170,
      70176,
      70182,
      70188,
      70194,
      70200,
      70206,
      70212,
      70218,
      70224
    ],
    "core.event_count": 19,
    "core.first_event_number": 70116,
    "core.last_event_number": 70224,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0615_df-s02-d0_dw_0_20260203T223551.hdf5"
    }
  ]
}
justIN time: 2026-02-04 09:15:21 UTC       justIN version: 01.06.00