Love Fuel?    Donate

FuelPHP Forums

Ask your question about FuelPHP in the appropriate forum, or help others by answering their questions.
Memory consumption is very high when ORM is used inside loops
  • Hello there,

    Sorry for ask you this kind of question.

    But, I have memory leak when I have ORM inside loop like following.

    $arr = array(0=>'a,c,d,e,f,g,h')  // I have more than 1000000 records in this array
                                                             // its value is comma separated data

    foreach($arr as $key => $value)
    {
    $vs = explode(',',$value);
    foreach($vs as $no => $data)
    {
    if(\Model_A::isRegister($data))
    {
    \Model_A::register($data);
    \Model_B::register($data);
    }
    }
    }

    I use memory_get_usage() and look at memory usage and it showed memory consumption is very high. 
    Therefore, the program is stopped.

    Do you have any suggestion?

    I tried to replace all find method to DB::select().
    But, still memory consumption is very high and stopped the program.

    PHP version 7.0
    Ubuntu: 16.04
  • There is no memory leak, ORM objects are cached by default to ensure data consistency. This makes sure that if you run two queries that result in the same record, those will be represented by the same object.

    Assuming your isRegister() method fetches an object, your loop will try to put more than 1000000 record objects in memory. So it runs out.

    You can disable ORM object caching by adding:

    ->from_cache(false)->

    to your query. Not that if you don't delete the variable holding the result, PHP will still have the object in memory. 

    Or by calling:

    ModelName::flush_cache('\This\Model");

    which will flush all \This\Model objects from the ORM cache. 

    DB:select() results aren't cached by the framework, so if memory usage goes op in a loop, you are not flushing the results (i.e. destroying the variable holding the result of the execute() method).
  • Thanks for your reply.

    Possibly, I am not flushing object correctly.
    In fact, I did Model_A::flush_cache().
    But, it did not work.
    The reason why it did not work is maybe class name is missing in the parameter.

    I will take a look tomorrow morning.
  • Hello there,

    I have lots of method like following and eating up my memory.

    public static function isRegister(int $user_id):bool
    {
      $ret = \DB::select('id')->from(self::$_table_name)
            ->where('user_id','=',$user_id)
            ->and_where('current_position','=','1')
            ->and_where('status','=','1')
            ->execute();
      if(isset($ret[0]))
      {
        unset($ret);
        return true;
      }
      unset($ret);
      return false;
    }

    Do you have any suggestion for this?
  • If you don't pass an argument to flush_cache(), it will use the name of the class it's called on, in your case Model_A.


  • $ret is a local variable, so PHP will delete that as soon as the method call ends. So there is no memory leak, memory use is temporary.

    Obviously, you are selecting data, which will be fetched all into memory and returned in a "cached" object. If you are selecting 100.000 records, you need memory for that many records.

    If you don't want to fetch them into memory but iterate over the result only once, you can include

    ->caching(false)->

    in a DB query, which will return a simple result object instead of a cached object. Result objects will fetch one record at the time when you iterate over it. 

    The data will still be cached, but now by the php database driver and not by the app (so the actual memory usage won't be that different). 

    It is similar to coding a fetch loop over a result object, with the same limitations. Once fetched it's gone from the queue, and you can only fetch sequentially, there is no random access.
  • Thank you for your explanation.

    For the DB class, I guess following correct?

    ->cached(0,null,false)->


  • Careful, there is a big difference between

    caching()

    which determines if you get a Cached object back (with all results) or a Result object (which you can only interate over sequentionally, and fetches one record at the time while you do)

    and 

    cache()

    which uses Fuel's Cache class to cache results in your configured cache backend.
  • Thanks for your suggestion.

    When I looked at \Database_Query class, caching method dose not exist.

    So, we might need like following method.

      /**
       * Enables or disables the query to be cached.
       *
       * @param   boolean $cache_all if true, cache all results, even empty ones
       *
       * @return  $this
       */
      public function caching($cache_all = true)
      {
        $this->_cache_all = (bool) $cache_all;
        return $this;
      }

    Is that technically okay ?
  • I've looked in the code too, it's been a while since I've used this. ;-)

    caching() is a method on the connection (so in Database_Connection), not on the query. It sets a config, and acts on all queries using that connection from that point on. 

    NB: also on queries generated by ORM, so you might see a performance impact there).
  • So, that would be like following?

    \Database_Connection::instance()->caching(false)?

    I saw it enables and disables cache config.

    Then, execute() method dose not give me the result of query and it is like following.

    Fuel\Core\Database_PDO_Result Object
    (
        [_query:protected] => SELECT `id` FROM `model` WHERE `tel` = '09000000000'
        [_result:protected] => PDOStatement Object
            (
                [queryString] => SELECT `id` FROM `model` WHERE `tel` = '09000000000'
            )

        [_results:protected] => 
        [_row:protected] => 
        [_total_rows:protected] => 1
        [_current_row:protected] => -1
        [_as_object:protected] => 
        [_sanitization_enabled:protected] => 
    )
  • Harro VertonHarro Verton
    Accepted Answer
    As I said, there are two different result objects:

    1. Database_xxx_Cached

    This will fetch ALL rows of the result, and stores it in the object. This will allow random access on key, will allow non-sequential access, and will allow multiple access as all rows will be there until you destroy the object. Obviously, this requires the memory to store all those rows.

    2. Database_xxx_Result

    This is an iterator. It does NOT fetch ALL rows, it will fetch one row at the time, and only the moment you access it. It will not store fetched rows. So it has a very low memory footprint, but you can only iterate over it. There is no random access (as there is nothing stored to access), you can iterate only once (once it is fetched it is gone from the database driver).

    In plain PHP, tou can roughly compare the functionality to the difference between:

    $allrows = $pdoresult->FetchAll();
    foreach ($allrows as $row)
    {
        // process the row
    }

    vs.

    while ($row = $pdoresult->fetch(PDO::FETCH_ASSOC))
    {
        // process the row
    }

  • Just committed an update to 1.9/dev that allows you to choose the result object type per query:

    $result = DB::select()->from('table')->caching(false)->execute();

    instead of having to set it globally (and not forget to change it back again after you've set it.
  • Thank you for your detailed information and documentation.

    This helps me a lot.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

In this Discussion