I've written a controller and the index action serves page with a datatable when loaded in browser and returns the datatable elements JSON when reached with ajax request.
My question is: which above mentioned function is better to use?
From a best practice point of view, your REST API should be separate from your interactive pages. So I would not build something like this.
Also, you can only do this with Controller_Hybrid, and as of 1.7.1 is can handle this itself (although not with a datatable but with a JSON dump), but not when in production (it is considered a security risk if your API responses can be dumped into a browser).
I consider it bad practice if a REST API returns data on a browser request. I can understand it's useful for development purposes, so Controller_Rest (as of 1.7.1) will dump an array returned by an action method as a JSON "printout" if the requested format is not compatible (notably "html", the default of a browser request) when not running in a production environment, and will return an error (406 NOT ACCEPTABLE) when in production.
If your application has a functional requirement to provide the same data both through an API and through a web page, best practices dictate they should be two different functions, in two different controllers.