justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 247218.134@dunegpschedd02.fnal.gov

Jobsub ID247218.134@dunegpschedd02.fnal.gov
Workflow ID10258
Stage ID1
User nameepennacc@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-16 20:25:17
SiteUK_Edinburgh
EntryDUNE_UK_SGridECDF_ce1_multicore
Last heartbeat2025-11-16 20:38:23
From worker nodeHostnamenode2b03.ecdf.ed.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-16 20:27:13
Input filesfardet-hd:prodgenie_nue_dune10kt_1x2x6_20251003T230533Z_gen_000465_g4_detsim.root
fardet-hd:prodgenie_nue_dune10kt_1x2x6_20251004T012412Z_gen_003623_g4_detsim.root
fardet-hd:prodgenie_nue_dune10kt_1x2x6_20251004T021505Z_gen_005089_g4_detsim.root
fardet-hd:prodgenie_nue_dune10kt_1x2x6_20251003T223137Z_gen_000051_g4_detsim.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-16 20:38:23
Saved logsjustin-logs:247218.134-dunegpschedd02.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

[ERROR] Socket error
[2025-11-16 20:37:56.803216 +0000][Debug  ][AsyncSock         ] [ccdcacli447.in2p3.fr:30719.0] Closing the socket
[2025-11-16 20:37:56.803224 +0000][Debug  ][Poller            ] <[::ffff:192.41.105.36]:51128><--><[::ffff:134.158.209.218]:30719> Removing socket from the poller
[2025-11-16 20:37:56.803277 +0000][Debug  ][PostMaster        ] [ccdcacli447.in2p3.fr:30719] Recovering error for stream #0: [ERROR] Socket error.
[2025-11-16 20:37:56.803301 +0000][Debug  ][PostMaster        ] [ccdcacli447.in2p3.fr:30719] Reporting disconnection to queued message handlers.
[2025-11-16 20:37:56.803308 +0000][Debug  ][XRootD            ] [ccdcacli447.in2p3.fr:30719] Handling error while processing kXR_read (handle: 0x00000000, offset: 50275472, size: 179547277): [ERROR] Socket error.
[2025-11-16 20:37:56.803313 +0000][Error  ][XRootD            ] [ccdcacli447.in2p3.fr:30719] Unable to get the response to request kXR_read (handle: 0x00000000, offset: 50275472, size: 179547277)
[2025-11-16 20:37:56.803318 +0000][Debug  ][ExDbgMsg          ] [ccdcacli447.in2p3.fr:30719] Passing to the thread-pool MsgHandler: 0xa1c24c0 (message: kXR_read (handle: 0x00000000, offset: 50275472, size: 179547277) ).
[2025-11-16 20:37:56.803342 +0000][Debug  ][ExDbgMsg          ] [ccdcacli447.in2p3.fr:30719] Calling MsgHandler: 0xa1c24c0 (message: kXR_read (handle: 0x00000000, offset: 50275472, size: 179547277) ) with status: [ERROR] Socket error.
[2025-11-16 20:37:56.803354 +0000][Debug  ][File              ] [0x777c770@root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/fb/9a/prodgenie_nue_dune10kt_1x2x6_20251004T012412Z_gen_003623_g4_detsim.root?xrdcl.requuid=d8df1a12-9688-409d-99fb-9f559e3acd3c] Running the recovery procedure
[2025-11-16 20:37:56.803381 +0000][Debug  ][ExDbgMsg          ] [ccxrootdegee.in2p3.fr:1094] MsgHandler created: 0xa1bc5a0 (message: kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/fb/9a/prodgenie_nue_dune10kt_1x2x6_20251004T012412Z_gen_003623_g4_detsim.root, mode: 00, flags: kXR_open_read ) ).
[2025-11-16 20:37:56.803397 +0000][Debug  ][ExDbgMsg          ] [ccdcacli447.in2p3.fr:30719] Destroying MsgHandler: 0xa1c24c0.
[2025-11-16 20:37:56.803445 +0000][Debug  ][ExDbgMsg          ] [ccxrootdegee.in2p3.fr:1094] Moving MsgHandler: 0xa1bc5a0 (message: kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/fb/9a/prodgenie_nue_dune10kt_1x2x6_20251004T012412Z_gen_003623_g4_detsim.root, mode: 00, flags: kXR_open_read ) ) from out-queu to in-queue.
[2025-11-16 20:37:59.792092 +0000][Debug  ][ExDbgMsg          ] [msg: 0xa1bf590] Assigned MsgHandler: 0xa1bc5a0.
[2025-11-16 20:37:59.792138 +0000][Debug  ][ExDbgMsg          ] [handler: 0xa1bc5a0] Removed MsgHandler: 0xa1bc5a0 from the in-queue.
[2025-11-16 20:37:59.792193 +0000][Debug  ][XRootD            ] [ccxrootdegee.in2p3.fr:1094] Handling error while processing kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/fb/9a/prodgenie_nue_dune10kt_1x2x6_20251004T012412Z_gen_003623_g4_detsim.root, mode: 00, flags: kXR_open_read ): [ERROR] Error response: bad address.
[2025-11-16 20:37:59.792678 +0000][Debug  ][ExDbgMsg          ] [ccxrootdegee.in2p3.fr:1094] Calling MsgHandler: 0xa1bc5a0 (message: kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/fb/9a/prodgenie_nue_dune10kt_1x2x6_20251004T012412Z_gen_003623_g4_detsim.root, mode: 00, flags: kXR_open_read ) ) with status: [ERROR] Error response: bad address.
[2025-11-16 20:37:59.792768 +0000][Debug  ][File              ] [0x777c770@root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/fb/9a/prodgenie_nue_dune10kt_1x2x6_20251004T012412Z_gen_003623_g4_detsim.root?xrdcl.requuid=d8df1a12-9688-409d-99fb-9f559e3acd3c] Open has returned with status [ERROR] Server responded with an error: [3012] Internal timeout
[2025-11-16 20:37:59.792779 +0000][Debug  ][File              ] [0x777c770@root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/fb/9a/prodgenie_nue_dune10kt_1x2x6_20251004T012412Z_gen_003623_g4_detsim.root?xrdcl.requuid=d8df1a12-9688-409d-99fb-9f559e3acd3c] Error while opening at ccxrootdegee.in2p3.fr:1094: [ERROR] Server responded with an error: [3012] Internal timeout
[2025-11-16 20:37:59.792851 +0000][Debug  ][ExDbgMsg          ] [ccxrootdegee.in2p3.fr:1094] Destroying MsgHandler: 0xa1bc5a0.
16-Nov-2025 20:37:59 GMT  Opened output file with pattern "prodgenie_nue_dune10kt_1x2x6_20251004T012412Z_gen_003623_g4_detsim_20251116T203139Z_reco.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
[ No processed events ]
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1402.02 MB
  Peak resident set size usage (VmHWM): 670.519 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 16-Nov-2025 20:38:00 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- FileReadError BEGIN
      ---- FatalRootError BEGIN
        Fatal Root Error: TNetXNGFile::ReadBuffer
        [ERROR] Server responded with an error: [3012] Internal timeout
        ROOT severity: 3000
      ---- FatalRootError END
      
      The above exception was thrown while processing module TriggerPrimitiveMakerTPC/tpmakerTPCsimpleThr run: 8528 subRun: 0 event: 36221
    ---- FileReadError END
    Exception going through path makers
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::TNetXNGFile
  The remote file is not open
  ROOT severity: 3000
---- FatalRootError END
---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Server responded with an error: [3012] Internal timeout
  ROOT severity: 3000
---- FatalRootError END
---- FatalRootError BEGIN
  Fatal Root Error: TNetXNGFile::Close
  [ERROR] Server responded with an error: [3012] Internal timeout
  ROOT severity: 3000
---- FatalRootError END
%MSG
[2025-11-16 20:38:00.547599 +0000][Debug  ][JobMgr            ] Stopping the job manager...
[2025-11-16 20:38:00.548110 +0000][Debug  ][JobMgr            ] Job manager stopped
[2025-11-16 20:38:00.548165 +0000][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-11-16 20:38:00.548245 +0000][Debug  ][TaskMgr           ] Task manager stopped
[2025-11-16 20:38:00.548251 +0000][Debug  ][Poller            ] Stopping the poller...
[2025-11-16 20:38:00.548506 +0000][Debug  ][AsyncSock         ] [ccdcacli447.in2p3.fr:30033.0] Closing the socket
[2025-11-16 20:38:00.548518 +0000][Debug  ][PostMaster        ] [ccdcacli447.in2p3.fr:30033] Destroying stream
[2025-11-16 20:38:00.548555 +0000][Debug  ][AsyncSock         ] [ccdcacli447.in2p3.fr:30033.0] Closing the socket
[2025-11-16 20:38:00.548595 +0000][Debug  ][AsyncSock         ] [ccdcacli447.in2p3.fr:30463.0] Closing the socket
[2025-11-16 20:38:00.548604 +0000][Debug  ][PostMaster        ] [ccdcacli447.in2p3.fr:30463] Destroying stream
[2025-11-16 20:38:00.548608 +0000][Debug  ][AsyncSock         ] [ccdcacli447.in2p3.fr:30463.0] Closing the socket
[2025-11-16 20:38:00.548623 +0000][Debug  ][AsyncSock         ] [ccdcacli447.in2p3.fr:30562.0] Closing the socket
[2025-11-16 20:38:00.548629 +0000][Debug  ][PostMaster        ] [ccdcacli447.in2p3.fr:30562] Destroying stream
[2025-11-16 20:38:00.548632 +0000][Debug  ][AsyncSock         ] [ccdcacli447.in2p3.fr:30562.0] Closing the socket
[2025-11-16 20:38:00.548643 +0000][Debug  ][AsyncSock         ] [ccdcacli447.in2p3.fr:30655.0] Closing the socket
[2025-11-16 20:38:00.548649 +0000][Debug  ][PostMaster        ] [ccdcacli447.in2p3.fr:30655] Destroying stream
[2025-11-16 20:38:00.548653 +0000][Debug  ][AsyncSock         ] [ccdcacli447.in2p3.fr:30655.0] Closing the socket
[2025-11-16 20:38:00.548663 +0000][Debug  ][AsyncSock         ] [ccdcacli447.in2p3.fr:30719.0] Closing the socket
[2025-11-16 20:38:00.548667 +0000][Debug  ][PostMaster        ] [ccdcacli447.in2p3.fr:30719] Destroying stream
[2025-11-16 20:38:00.548842 +0000][Debug  ][AsyncSock         ] [ccdcacli447.in2p3.fr:30719.0] Closing the socket
[2025-11-16 20:38:00.548854 +0000][Debug  ][AsyncSock         ] [ccdcacli447.in2p3.fr:30777.0] Closing the socket
[2025-11-16 20:38:00.548860 +0000][Debug  ][PostMaster        ] [ccdcacli447.in2p3.fr:30777] Destroying stream
[2025-11-16 20:38:00.548864 +0000][Debug  ][AsyncSock         ] [ccdcacli447.in2p3.fr:30777.0] Closing the socket
[2025-11-16 20:38:00.548873 +0000][Debug  ][AsyncSock         ] [ccdcacli447.in2p3.fr:30833.0] Closing the socket
[2025-11-16 20:38:00.548881 +0000][Debug  ][PostMaster        ] [ccdcacli447.in2p3.fr:30833] Destroying stream
[2025-11-16 20:38:00.548884 +0000][Debug  ][AsyncSock         ] [ccdcacli447.in2p3.fr:30833.0] Closing the socket
[2025-11-16 20:38:00.548894 +0000][Debug  ][AsyncSock         ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket
[2025-11-16 20:38:00.548902 +0000][Debug  ][Poller            ] <[::ffff:192.41.105.36]:40314><--><[::ffff:134.158.209.218]:1094> Removing socket from the poller
[2025-11-16 20:38:00.549192 +0000][Debug  ][PostMaster        ] [ccxrootdegee.in2p3.fr:1094] Destroying stream
[2025-11-16 20:38:00.549210 +0000][Debug  ][AsyncSock         ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket
Art has completed and will exit with status 1.
justIN time: 2025-12-19 12:14:20 UTC       justIN version: 01.05.03