HTML5 has <video/> element that downloads video from server, decodes it and renders. Often if not always they use hardware accelerated decoding (if available).
Is it possible to access just decoding functionality? The reason is that I'm using custom streaming protocol and so on client side I have encoded video stream that I need to decode and render.
Pure JavaScript implementations of video decoder are not applicable unfortunately as they cannot provide sufficient performance. I'm interested in HVEC or h.264 codecs only.
Is it possible to access just decoding functionality?
Unfortunately no. We only have access to a high-level API dealing with the stream/source file agnostically with limited influence such as time-based position, play state and various events. We can draw frames to a canvas as raw RGB(A) from the current decoded frame, but that's about it.
The reason is that I'm using custom streaming protocol and so on
client side I have encoded video stream that I need to decode and
render
You aren't describing this protocol so we can only guess, but you might be able to build a browser compatible stream that can be used by the video element using Media Source Extensions. This allows you to build adaptive and custom streaming solutions right in the client.
Pure JavaScript implementations of video decoder are not applicable
unfortunately as they cannot provide sufficient performance.
This is not necessarily true. Examples are for example pure JS implementation that decodes MPEG1 streams in real-time, both audio and video, such as this and this. Of course, this works on the very limit of what most browsers can do at present time. There is also an emscripten based H-264 decoder available that seem to also utilize the GPU via WebGL, but I cannot speak for its performance - it may be a good starting point for the next paragraph though:
A better option is to look into WebAssembly which can run pre-compiled binary code from for example C/C++ source-code. This allows you to use open-source implementations of HVEC/H.264 decoders running at native speed in the browser (careful with the licenses and terms though, especially for H.264), or use parts of software such as (linkable) ffmpeg.
I'm interested in any even non-portable solution
You may in that case want to look into building a web-extension (aka browser extensions) which can use messaging to interact with native application (the latter could be ffmpeg in this case, or a program that can deal with the stream directly).
How exactly this would work will of course depend on the protocol you're using and so forth.
Just my 2 cents based on the limited scope/description.
The solution for this problem is WebRTC. It's possible to integrate an external encoder or to use an embedded one. In browser WebRTC client uses H/W decoding. WebRTC also provides real-time streaming functionality. Compatibility is not bad.
After long research, decoding h264 HLS TS segment stream using hardware decoder is possible on Android browsers using media source extension (MSE). Since MSE is not supported by iOS, getting it to work on Safari in iOS seems like hit a roadblock as Apple doesn't allow access to hardware decoder via FIFO buffer or callbacks. Given, Apple's support to WebRTC, seems like only way to get to hardware decoder in iOS would be equivalent to "receiving a video call" flow except the input has to be a remote http stream and output has to go to canvas.
