Collection walk iterator magento

In this article I will explain you how to efficiently read and update thousands of records in magento, so that when your command object is running it does not run in "Allowed memory size of # bytes exhausted" problem.

There are several ways to eradicate this problem. The approach we are taking uses the class Magento\Framework\Model\ResourceModel\Iterator to efficiently use the memory while several records are read and updated in a loop.

The idea here is to only ever load a single row into memory at any time. By reading only single row at a time, it does not matter how many rows we need to process, it should not increase our memory consumption.

use Magento\Framework\Model\ResourceModel\Iterator;
use Magento\Framework\Model\ResourceModel\IteratorFactory;
use Magento\Sales\Model\ResourceModel\Order\CollectionFactoryInterface;
use Zend_Db_Select;

class RunSomeLongTask
{
    public const ORDER_STATE_VALUE = 'example';

    /** @var CollectionFactoryInterface  */
    private $orderCollectionFactory;

    /** @var IteratorFactory */
    private $iteratorFactory;

    /** @var SomeTask */
    private $someTask;

    /**
     * RunSomeLongTask constructor.
     * @param CollectionFactoryInterface $orderCollectionFactory
     * @param IteratorFactory $iteratorFactory
     * @param SomeTask $someTask
     */
    public function __construct(
        CollectionFactoryInterface $orderCollectionFactory,
        IteratorFactory $iteratorFactory,
        SomeTask $someTask
    ) {
        $this->orderCollectionFactory = $orderCollectionFactory;
        $this->iteratorFactory = $iteratorFactory;
        $this->someTask = $someTask; // "important: $iterator->walk will send the result of each row to this class, function run"
    }

    public function process(): void
    {
        $orderCollection = $this->orderCollectionFactory->create();

	    // only fetch the columns you need
        $orderCollection
            ->getSelect()
            ->reset(Zend_Db_Select::COLUMNS)
            ->columns(['entity_id', 'state', 'status'])
            ->where('state=?', self::ORDER_STATE_VALUE);

        /** @var Iterator $iterator */
        $iterator = $this->iteratorFactory->create();

        $iterator->walk($orderCollection->getSelect(), [[
            $this->someTask, // instance of class SomeTask
            'run' // the function in your class "SomeTask"
        ]]);
    }
}


class SomeTask
{
    public function run(array $args): void
    {
        // you will receive the row data here
        $orderId = $args['row']['entity_id'];

        // do your task for example (call another api, update another model from magento etc
        $this->restApi->updateOrder($orderId); // this line is just an example
    }
}

Why OFFSET/LIMIT is not a good option?

The approach of using OFFSET/LIMIT at database level is very inefficient. If we issed a query such as SELECT name FROM products OFFSET 5000 LIMIT 1000 we are basically telling the database to fetch 6000 records and throw the first 5000 away.

what clients say about us

Client testimonials

Our sites look great and run smoothly. Thanks for all your hard work and patience in getting us to where we are today.

contact us

Contact Us

If you have questions or want us to send you a quote for project please email us.

latest news

Articles

PHPUnit mock magic functions
Learn more about PHPUnit Partial Mock.